Oct 30 19:29:32.253633 ip-10-0-141-19 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:29:32.571586 ip-10-0-141-19 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:29:32.571586 ip-10-0-141-19 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:29:32.571586 ip-10-0-141-19 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:29:32.571586 ip-10-0-141-19 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:29:32.578575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578478    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:29:32.578575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578564    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:29:32.578575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578570    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:29:32.578575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578574    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:29:32.578575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578579    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:29:32.578575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578584    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578589    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578594    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578599    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578603    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578607    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578611    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578615    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578619    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578623    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578627    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578631    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578634    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578638    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578642    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578645    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578649    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578652    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578657    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578660    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:29:32.578922 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578664    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578668    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578672    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578676    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578680    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578684    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578687    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578699    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578704    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578709    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578714    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578718    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578722    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578726    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578731    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578735    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578740    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578744    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578748    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:29:32.579705 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578753    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578758    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578762    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578766    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578770    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578775    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578788    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578793    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578797    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578802    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578806    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578811    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578815    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578819    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578824    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578828    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578844    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578849    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578853    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:29:32.580193 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578857    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578861    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578865    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578870    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578874    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578878    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578883    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578887    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578891    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578896    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578899    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578903    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578908    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578912    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578916    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578920    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578924    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578927    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578932    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:29:32.580658 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578939    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578947    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578952    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578957    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578961    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578965    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578970    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578974    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578978    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578985    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578993    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.578999    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.579003    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.579007    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.579011    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.579016    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.579021    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.579025    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.579030    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.579035    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:29:32.581532 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.579039    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.579043    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.579047    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.579053    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581218    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581231    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581236    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581240    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581244    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581248    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581252    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581256    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581260    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581265    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581270    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581274    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581278    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581282    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581287    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581291    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:29:32.582425 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581295    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:29:32.583265 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581299    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:29:32.583265 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581303    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:29:32.583265 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581308    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:29:32.583265 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581312    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:29:32.583265 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581316    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:29:32.583265 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581320    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:29:32.583265 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581324    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:29:32.583265 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581329    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:29:32.583265 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581335    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:29:32.583265 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581342    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:29:32.583265 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581346    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:29:32.583265 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581351    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:29:32.583265 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581356    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:29:32.583265 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581379    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:29:32.583265 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581385    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:29:32.583265 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581389    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:29:32.583265 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581393    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:29:32.583265 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581397    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581402    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581406    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581410    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581414    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581419    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581423    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581427    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581431    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581438    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581445    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581451    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581456    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581460    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581464    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581469    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581473    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581477    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581481    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581485    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:29:32.583726 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581489    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581493    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581497    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581502    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581506    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581510    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581515    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581519    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581523    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581527    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581531    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581535    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581539    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581542    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581546    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581550    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581554    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581558    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581562    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581566    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:29:32.584207 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581571    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581575    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581580    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581584    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581597    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581602    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581606    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581610    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581614    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581618    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581641    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581645    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581650    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581654    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581659    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581663    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581667    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581674    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581679    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:29:32.584699 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581683    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581687    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581691    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581696    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581700    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581704    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581708    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581712    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581717    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581722    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581726    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581731    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.581735    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582624    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582635    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582640    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582646    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582650    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582655    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582660    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:29:32.585430 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582664    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582669    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582673    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582677    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582681    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582685    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582690    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582694    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582698    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582703    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582707    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582711    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582716    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582720    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582724    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582729    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582733    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582737    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582741    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582746    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:29:32.586969 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582751    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:29:32.587554 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582755    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:29:32.587554 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582760    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:29:32.587554 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582764    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:29:32.587554 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582769    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:29:32.587554 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582773    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:29:32.587554 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582777    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:29:32.587554 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582782    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:29:32.587554 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582787    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:29:32.587554 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582792    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:29:32.587554 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582799    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:29:32.587554 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582805    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:29:32.587554 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582809    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:29:32.587554 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582814    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:29:32.587554 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582819    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:29:32.587554 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582824    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:29:32.587554 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582839    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:29:32.587554 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582843    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:29:32.587554 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582847    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582852    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582856    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582860    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582863    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582871    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582877    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582882    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582887    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582892    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582897    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582901    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582905    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582910    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582914    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582918    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582922    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582926    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582931    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:29:32.588094 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582935    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582939    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582943    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582947    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582952    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582956    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582961    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582965    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582969    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582973    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582977    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582981    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582985    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582989    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582993    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.582998    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583001    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583005    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583010    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583014    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:29:32.588588 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583018    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583022    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583027    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583031    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583035    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583040    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583045    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583049    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583053    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583057    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583061    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583065    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583069    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583074    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583078    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583082    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583086    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583091    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583095    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583099    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:29:32.589141 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583103    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.583107    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584807    2577 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584826    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584839    2577 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584846    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584854    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584859    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584870    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584877    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584882    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584887    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584893    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584898    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584903    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584907    2577 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584912    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584917    2577 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584922    2577 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584926    2577 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584932    2577 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584937    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584942    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584947    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:29:32.589665 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584952    2577 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584959    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584963    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584968    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584973    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584978    2577 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584984    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584989    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584994    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.584998    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585005    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585010    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585014    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585019    2577 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585024    2577 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585029    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585036    2577 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585041    2577 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585046    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585050    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585055    2577 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585062    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585066    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585071    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585076    2577 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:29:32.590213 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585081    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585085    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585090    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585094    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585099    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585104    2577 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585108    2577 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585115    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585120    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585125    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585133    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585139    2577 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585143    2577 flags.go:64] FLAG: --help="false"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585148    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585154    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585159    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585164    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585169    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585175    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585180    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585185    2577 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585190    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585194    2577 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:29:32.590867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585199    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585204    2577 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585208    2577 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585213    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585218    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585223    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585227    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585232    2577 flags.go:64] FLAG: --lock-file=""
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585236    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585241    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585247    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585255    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585260    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585265    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585270    2577 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585274    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585280    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585284    2577 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585289    2577 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585298    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585304    2577 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585310    2577 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585315    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585320    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585325    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:29:32.591415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585330    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585334    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585338    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585343    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585354    2577 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585374    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585379    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585383    2577 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585388    2577 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585393    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585398    2577 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585403    2577 flags.go:64] FLAG: --port="10250"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585407    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585412    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-09ea4d04e90b55a23"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585417    2577 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585422    2577 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585426    2577 flags.go:64] FLAG: --register-node="true"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585431    2577 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585437    2577 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585441    2577 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585445    2577 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585449    2577 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585456    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585461    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585465    2577 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585470    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:29:32.592660 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585479    2577 flags.go:64] FLAG: --runonce="false"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585484    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585489    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585494    2577 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585498    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585502    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585510    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585517    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585522    2577 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585527    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585532    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585537    2577 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585541    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585546    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585551    2577 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585555    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585564    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585569    2577 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585573    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585580    2577 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585584    2577 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585596    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585600    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585604    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585607    2577 flags.go:64] FLAG: --v="2"
Oct 30 19:29:32.593295 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585612    2577 flags.go:64] FLAG: --version="false"
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585616    2577 flags.go:64] FLAG: --vmodule=""
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585620    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.585642    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585787    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585791    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585794    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585799    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585804    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585808    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585812    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585815    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585817    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585820    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585825    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585828    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585831    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585833    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585836    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:29:32.593923 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585838    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585841    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585844    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585847    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585850    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585852    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585855    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585857    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585860    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585862    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585865    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585868    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585871    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585874    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585876    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585879    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585881    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585884    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585886    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585889    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:29:32.594401 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585892    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585895    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585898    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585900    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585903    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585905    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585908    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585911    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585915    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585917    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585920    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585923    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585925    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585928    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585931    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585933    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585936    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585938    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585941    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585943    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:29:32.594892 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585946    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585948    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585951    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585954    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585956    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585959    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585961    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585964    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585966    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585969    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585971    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585974    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585977    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585980    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585982    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585985    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585987    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585990    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585992    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.585996    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:29:32.595343 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586000    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586011    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586015    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586018    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586021    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586024    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586026    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586029    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586031    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586034    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586037    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586039    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586042    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586045    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586047    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586050    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586052    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586055    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586058    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586061    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:29:32.595817 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586063    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:29:32.596264 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586066    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:29:32.596264 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586068    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:29:32.596264 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586071    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:29:32.596264 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586075    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:29:32.596264 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586077    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:29:32.596264 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586080    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:29:32.596264 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586082    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:29:32.596264 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586085    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:29:32.596264 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586088    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:29:32.596264 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586090    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:29:32.596264 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.586119    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:29:32.596264 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586158    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:29:32.596264 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586163    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:29:32.596264 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586167    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:29:32.596264 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586171    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586173    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586176    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586179    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586182    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586185    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586187    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586190    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586193    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586196    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586198    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586201    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586204    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586207    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586209    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586212    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586215    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586217    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586220    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:29:32.596627 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586223    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586226    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586230    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586232    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586235    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586237    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586240    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586244    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586247    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586250    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586254    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586257    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586260    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586263    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586266    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586268    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586271    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586273    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586276    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:29:32.597070 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586279    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586282    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586284    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586287    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586290    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586292    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586295    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586297    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586300    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586302    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586305    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586308    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586310    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586313    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586316    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586319    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586322    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586324    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586327    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586329    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:29:32.597575 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586332    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586334    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586337    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586341    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586343    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586346    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586349    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586352    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586355    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586372    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586377    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586381    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586385    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586388    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586392    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586396    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586400    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586405    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586409    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586413    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:29:32.598073 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586417    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586421    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586425    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586429    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586433    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586437    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586441    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586446    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586451    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586455    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586458    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586462    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586466    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586469    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586473    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586479    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586483    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586488    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586492    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586498    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:29:32.598600 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586505    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586510    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586516    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586521    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.586525    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.594215    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.594234    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.594251    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.594256    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594312    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594317    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594321    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594324    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594328    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594331    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594333    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594336    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594339    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594342    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594344    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:29:32.599069 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594347    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594350    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594353    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594356    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594369    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594372    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594375    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594378    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594381    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594384    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594386    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594389    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594392    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594394    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594397    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594400    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594402    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594405    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594409    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594412    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:29:32.599545 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594415    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594418    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594421    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594424    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594427    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594430    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594433    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594435    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594438    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594441    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594443    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594446    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594450    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594453    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594456    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594459    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594462    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594465    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594468    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:29:32.599988 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594471    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594473    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594477    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594481    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594486    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594489    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594492    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594494    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594497    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594500    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594503    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594505    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594508    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594511    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594514    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594517    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594519    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594522    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594524    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594527    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:29:32.600421 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594530    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594532    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594535    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594537    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594540    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594542    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594545    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594548    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594550    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594553    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594555    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594558    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594561    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594563    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594566    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594569    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594572    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594574    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594577    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:29:32.600869 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594579    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:29:32.601294 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594582    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:29:32.601294 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594585    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:29:32.601294 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594588    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:29:32.601294 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594590    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:29:32.601294 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594593    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:29:32.601294 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594600    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:29:32.601294 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594603    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:29:32.601294 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594605    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:29:32.601294 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594608    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:29:32.601294 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594610    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:29:32.601294 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594613    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:29:32.601294 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594616    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:29:32.601294 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594618    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:29:32.601294 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594621    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:29:32.601294 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594623    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:29:32.601294 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594626    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:29:32.601691 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.594654    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:29:32.601691 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594790    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:29:32.601691 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594795    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:29:32.601691 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594799    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:29:32.601691 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594802    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:29:32.601691 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594804    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:29:32.601691 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594808    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:29:32.601691 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594810    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:29:32.601691 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594813    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:29:32.601691 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594816    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:29:32.601691 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594818    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:29:32.601691 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594821    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:29:32.601691 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594824    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:29:32.601691 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594827    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:29:32.601691 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594830    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:29:32.601691 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594832    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594848    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594852    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594854    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594857    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594859    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594861    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594865    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594867    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594870    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594873    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594875    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594878    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594880    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594882    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594885    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594887    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594890    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594892    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:29:32.602084 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594895    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594898    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594900    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594904    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594907    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594909    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594912    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594914    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594916    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594919    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594922    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594924    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594927    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594929    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594931    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594934    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594936    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594939    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594941    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594943    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:29:32.602593 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594946    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594949    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594951    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594954    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594956    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594959    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594963    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594966    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594969    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594971    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594975    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594977    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594980    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594983    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594985    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594987    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594990    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594992    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594995    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:29:32.603161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594997    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.594999    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595002    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595005    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595007    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595009    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595013    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595016    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595019    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595021    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595024    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595026    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595029    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595031    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595034    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595037    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595039    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595041    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595044    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:29:32.603786 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595046    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:29:32.604232 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595049    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:29:32.604232 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595051    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:29:32.604232 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595053    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:29:32.604232 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595056    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:29:32.604232 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595058    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:29:32.604232 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595061    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:29:32.604232 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595064    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:29:32.604232 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595066    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:29:32.604232 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595069    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:29:32.604232 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595071    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:29:32.604232 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595074    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:29:32.604232 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595076    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:29:32.604232 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595078    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:29:32.604232 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595081    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:29:32.604232 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.595105    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:29:32.604232 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595133    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595136    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595139    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595141    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595144    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595147    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595149    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595152    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595154    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595157    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595159    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595161    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595164    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595167    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595170    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595172    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595175    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595177    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595180    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:29:32.604826 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595182    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595184    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595187    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595189    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595192    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595195    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595197    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595199    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595202    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595204    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595207    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595209    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595214    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595217    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595221    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595224    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595226    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595229    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595231    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:29:32.605426 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595234    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595237    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595240    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595242    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595244    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595247    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595249    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595252    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595255    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595258    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595260    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595263    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595266    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595269    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595272    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595274    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595277    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595279    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595282    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:29:32.605916 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595284    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595287    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595289    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595291    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595294    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595296    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595300    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595303    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595305    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595308    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595310    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595313    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595316    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595318    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595321    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595323    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595326    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595328    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595331    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595333    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:29:32.606354 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595336    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595338    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595341    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595344    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595346    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595349    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595351    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595354    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595356    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595372    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595374    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595377    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595379    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595382    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595384    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595387    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595389    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595392    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595395    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:29:32.606884 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595398    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:29:32.607431 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595400    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:29:32.607431 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595403    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:29:32.607431 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595405    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:29:32.607431 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595408    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:29:32.607431 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595410    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:29:32.607431 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595413    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:29:32.607431 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595416    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:29:32.607431 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595420    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:29:32.607431 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:32.595422    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:29:32.607431 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.595499    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:29:32.607431 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.598670    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 19:29:32.607431 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.599662    2577 server.go:1014] "Starting client certificate rotation"
Oct 30 19:29:32.607431 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.599819    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:29:32.607431 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.599859    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:29:32.607431 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.601873    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:29:32.607431 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.605253    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:29:32.614986 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.614970    2577 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:29:32.619784 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.619768    2577 log.go:25] "Validated CRI v1 image API"
Oct 30 19:29:32.620583 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.620558    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:29:32.623320 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.623301    2577 fs.go:135] Filesystem UUIDs: map[1f5c32e2-90df-4269-bedf-fadc30face1e:/dev/nvme0n1p3 31d6c66c-1be2-4143-969e-87a199824e2e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 19:29:32.623399 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.623319    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 19:29:32.628627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.628507    2577 manager.go:217] Machine: {Timestamp:2026-10-30 19:29:32.627025618 +0000 UTC m=+0.273333519 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3151783 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aaf5fc670c91356b37b6d6ac40f98 SystemUUID:ec2aaf5f-c670-c913-56b3-7b6d6ac40f98 BootID:e971d40e-99a9-4667-9f85-6e3c468425b7 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fd:5c:5d:b5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fd:5c:5d:b5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:94:fd:05:9b:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:29:32.628627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.628625    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:29:32.628741 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.628707    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:29:32.632169 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.632142    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:29:32.632314 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.632171    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-19.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:29:32.632369 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.632319    2577 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:29:32.632369 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.632327    2577 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:29:32.632369 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.632353    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:29:32.632453 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.632377    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:29:32.633661 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.633649    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:29:32.637352 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.637340    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:29:32.637482 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.637473    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:29:32.638217 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.638208    2577 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:29:32.638249 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.638223    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:29:32.638249 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.638234    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:29:32.638249 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.638243    2577 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:29:32.638389 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.638252    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:29:32.638708 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.638694    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:29:32.642392 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.642344    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:29:32.642962 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.642934    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:29:32.650520 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.650503    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:29:32.651494 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.651482    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:29:32.651531 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.651512    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:29:32.651679 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.651664    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:29:32.651706 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.651689    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:29:32.651706 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.651696    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:29:32.651706 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.651702    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:29:32.651790 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.651708    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:29:32.651790 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.651714    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:29:32.652581 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.652565    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:29:32.652581 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.652576    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:29:32.652581 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.652583    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:29:32.652714 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.652590    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:29:32.652714 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.652599    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:29:32.652714 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.652608    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:29:32.653355 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.653346    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:29:32.653355 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.653356    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:29:32.656892 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.656873    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-19.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 19:29:32.657700 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.657684    2577 server.go:1263] "Started kubelet"
Oct 30 19:29:32.657765 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.657736    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:29:32.658435 ip-10-0-141-19 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:29:32.659559 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.658645    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:29:32.659559 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.658720    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:29:32.660124 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.660107    2577 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:29:32.661219 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.660239    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:29:32.661272 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:32.661225    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-19.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 19:29:32.661272 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:32.661224    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 19:29:32.662696 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.661745    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6wz48"
Oct 30 19:29:32.664466 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:32.664445    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 19:29:32.668018 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.668003    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:29:32.668638 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.668618    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:29:32.669033 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:32.669014    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-19.us-east-2.compute.internal\" not found"
Oct 30 19:29:32.669529 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.669507    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:29:32.669529 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.669509    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:29:32.669529 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.669531    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:29:32.669690 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.669610    2577 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:29:32.669690 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.669618    2577 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:29:32.670326 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:32.670293    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-19.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 19:29:32.670433 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.670390    2577 factory.go:55] Registering systemd factory
Oct 30 19:29:32.670433 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.670406    2577 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:29:32.670702 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.670689    2577 factory.go:153] Registering CRI-O factory
Oct 30 19:29:32.670702 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.670702    2577 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:29:32.670783 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.670754    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:29:32.670783 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.670768    2577 factory.go:103] Registering Raw factory
Oct 30 19:29:32.670783 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.670777    2577 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:29:32.671485 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:32.670294    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-19.us-east-2.compute.internal.18e36567b181186e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-19.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-19.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:29:32.65765387 +0000 UTC m=+0.303961772,LastTimestamp:2026-10-30 19:29:32.65765387 +0000 UTC m=+0.303961772,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-19.us-east-2.compute.internal,}"
Oct 30 19:29:32.671819 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.671806    2577 manager.go:319] Starting recovery of all containers
Oct 30 19:29:32.676093 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:32.676054    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 19:29:32.677114 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.677094    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6wz48"
Oct 30 19:29:32.678973 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.678926    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:29:32.684909 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.684887    2577 manager.go:324] Recovery completed
Oct 30 19:29:32.686294 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:32.686277    2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 30 19:29:32.690648 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.690634    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:29:32.692488 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.692469    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:29:32.692567 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.692507    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:29:32.692567 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.692519    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:29:32.693033 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.693021    2577 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:29:32.693033 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.693028    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:29:32.693122 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.693033    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:29:32.693122 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.693051    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:29:32.693796 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.693779    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:29:32.693868 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.693810    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:29:32.693868 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.693822    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:29:32.695855 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:32.695769    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-19.us-east-2.compute.internal.18e36567b3949d8f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-19.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-19.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-19.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:29:32.692487567 +0000 UTC m=+0.338795476,LastTimestamp:2026-10-30 19:29:32.692487567 +0000 UTC m=+0.338795476,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-19.us-east-2.compute.internal,}"
Oct 30 19:29:32.769998 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:32.769969    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-19.us-east-2.compute.internal\" not found"
Oct 30 19:29:32.824186 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.824125    2577 policy_none.go:50] "Start"
Oct 30 19:29:32.824186 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.824165    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:29:32.824186 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.824181    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:29:32.825435 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.825420    2577 policy_none.go:44] "Start"
Oct 30 19:29:32.854928 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.854891    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:29:32.857069 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.857058    2577 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:29:32.857570 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:32.857553    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:29:32.857655 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.857573    2577 server.go:87] "Starting device plugin registration server"
Oct 30 19:29:32.858035 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.857867    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:29:32.858067 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.858045    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:29:32.858157 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.858142    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:29:32.858240 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.858224    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:29:32.858240 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.858235    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:29:32.858790 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:32.858771    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:29:32.858871 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:32.858814    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-19.us-east-2.compute.internal\" not found"
Oct 30 19:29:32.874941 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.874919    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:29:32.875024 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.874944    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:29:32.875024 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.874964    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:29:32.875024 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:32.875000    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 19:29:32.875705 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:32.875682    2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-19.us-east-2.compute.internal\" not found" node="ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:32.878024 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.878006    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:29:32.958702 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.958658    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:29:32.959656 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.959640    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:29:32.959756 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.959669    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:29:32.959756 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.959680    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:29:32.959756 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.959706    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:32.967947 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.967931    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:32.968013 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:32.967953    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-19.us-east-2.compute.internal\": node \"ip-10-0-141-19.us-east-2.compute.internal\" not found"
Oct 30 19:29:32.975090 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.975066    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal"]
Oct 30 19:29:32.975163 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.975134    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:29:32.975859 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.975845    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:29:32.975928 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.975870    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:29:32.975928 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.975882    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:29:32.977021 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.977009    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:29:32.977560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.977536    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:32.977683 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.977567    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:29:32.977745 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.977681    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:29:32.977745 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.977709    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:29:32.977745 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.977723    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:29:32.978310 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.978296    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:29:32.978401 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.978325    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:29:32.978401 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.978338    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:29:32.978887 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.978872    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:32.978963 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.978908    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:29:32.979901 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.979885    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:29:32.980000 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.979917    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:29:32.980000 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:32.979932    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:29:32.983711 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:32.983699    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-19.us-east-2.compute.internal\" not found"
Oct 30 19:29:33.000572 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:33.000556    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-19.us-east-2.compute.internal\" not found" node="ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:33.002598 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:33.002569    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-19.us-east-2.compute.internal\" not found" node="ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:33.071372 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:33.071336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4d61e23ca703103c56d5c7a000f30c8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal\" (UID: \"d4d61e23ca703103c56d5c7a000f30c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:33.071479 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:33.071377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4d61e23ca703103c56d5c7a000f30c8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal\" (UID: \"d4d61e23ca703103c56d5c7a000f30c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:33.071479 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:33.071400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dbf81fa0f84f4c1de7386fb2838c009-config\") pod \"kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal\" (UID: \"7dbf81fa0f84f4c1de7386fb2838c009\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:33.084657 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:33.084619    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-19.us-east-2.compute.internal\" not found"
Oct 30 19:29:33.172304 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:33.172281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4d61e23ca703103c56d5c7a000f30c8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal\" (UID: \"d4d61e23ca703103c56d5c7a000f30c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:33.172436 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:33.172308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4d61e23ca703103c56d5c7a000f30c8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal\" (UID: \"d4d61e23ca703103c56d5c7a000f30c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:33.172436 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:33.172333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dbf81fa0f84f4c1de7386fb2838c009-config\") pod \"kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal\" (UID: \"7dbf81fa0f84f4c1de7386fb2838c009\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:33.172436 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:33.172383    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4d61e23ca703103c56d5c7a000f30c8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal\" (UID: \"d4d61e23ca703103c56d5c7a000f30c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:33.172436 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:33.172400    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4d61e23ca703103c56d5c7a000f30c8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal\" (UID: \"d4d61e23ca703103c56d5c7a000f30c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:33.172436 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:33.172429    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dbf81fa0f84f4c1de7386fb2838c009-config\") pod \"kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal\" (UID: \"7dbf81fa0f84f4c1de7386fb2838c009\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:33.185396 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:33.185377    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-19.us-east-2.compute.internal\" not found"
Oct 30 19:29:33.285878 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:33.285849    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-19.us-east-2.compute.internal\" not found"
Oct 30 19:29:33.304021 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:33.304002    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:33.306095 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:33.306080    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:33.386598 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:33.386540    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-19.us-east-2.compute.internal\" not found"
Oct 30 19:29:33.487019 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:33.486984    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-19.us-east-2.compute.internal\" not found"
Oct 30 19:29:33.587559 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:33.587538    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-19.us-east-2.compute.internal\" not found"
Oct 30 19:29:33.600696 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:33.600679    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 19:29:33.600825 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:33.600809    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:29:33.687037 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:33.686962    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:24:33 +0000 UTC" deadline="2028-04-10 03:05:21.255868537 +0000 UTC"
Oct 30 19:29:33.687037 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:33.686999    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12655h35m47.568872024s"
Oct 30 19:29:33.687704 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:33.687691    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-19.us-east-2.compute.internal\" not found"
Oct 30 19:29:33.788213 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:33.788183    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-19.us-east-2.compute.internal\" not found"
Oct 30 19:29:33.889046 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:33.889026    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-19.us-east-2.compute.internal\" not found"
Oct 30 19:29:33.890529 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:33.890501    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbf81fa0f84f4c1de7386fb2838c009.slice/crio-f5b879a0b0afdc510709a56fedddbe3e63eee4eb9a2d869df77a7d9f3235cde6 WatchSource:0}: Error finding container f5b879a0b0afdc510709a56fedddbe3e63eee4eb9a2d869df77a7d9f3235cde6: Status 404 returned error can't find the container with id f5b879a0b0afdc510709a56fedddbe3e63eee4eb9a2d869df77a7d9f3235cde6
Oct 30 19:29:33.890744 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:33.890723    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d61e23ca703103c56d5c7a000f30c8.slice/crio-c3e72d26b223d07b1cab9286bc95e64e8f5540868af1598407589ef50ee834aa WatchSource:0}: Error finding container c3e72d26b223d07b1cab9286bc95e64e8f5540868af1598407589ef50ee834aa: Status 404 returned error can't find the container with id c3e72d26b223d07b1cab9286bc95e64e8f5540868af1598407589ef50ee834aa
Oct 30 19:29:33.895755 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:33.895734    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:29:35.172174 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.172133    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:29:35.201204 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.201183    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:35.216541 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.216524    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:29:35.217852 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.217839    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:29:35.228890 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.228875    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:29:35.278188 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.278172    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:29:35.318633 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.318611    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:29:35.871704 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.871678    2577 apiserver.go:52] "Watching apiserver"
Oct 30 19:29:35.881340 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.881274    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:29:35.881990 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.881655    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-d2hv2","kube-system/konnectivity-agent-kspfd","openshift-cluster-node-tuning-operator/tuned-bkdq5","openshift-dns/node-resolver-gmrbv","openshift-network-diagnostics/network-check-target-vwkv5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn","openshift-image-registry/node-ca-h5m52","openshift-multus/network-metrics-daemon-mwcmw","kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-qmqjk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal","openshift-multus/multus-d6gqm","openshift-network-operator/iptables-alerter-5n649","openshift-ovn-kubernetes/ovnkube-node-gjswr"]
Oct 30 19:29:35.883697 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.883184    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h5m52"
Oct 30 19:29:35.884381 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.884277    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5n649"
Oct 30 19:29:35.885470 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.885451    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:35.886668 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.886649    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gmrbv"
Oct 30 19:29:35.888265 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.888246    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:35.888375 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:35.888319    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:29:35.889894 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.889382    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:35.891098 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.890815    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.892470 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.892273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:35.892470 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:35.892330    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:29:35.892470 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.892400    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:29:35.893833 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.893015    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xs4jl\""
Oct 30 19:29:35.893833 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.893159    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zqbbj\""
Oct 30 19:29:35.893833 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.893321    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:29:35.893833 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.893557    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:29:35.893833 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.893716    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q2kw5\""
Oct 30 19:29:35.895330 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.894167    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:29:35.895330 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.894238    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:29:35.895330 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.894595    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:29:35.895330 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.892401    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:29:35.895330 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.894861    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:29:35.895330 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.895017    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pn76m\""
Oct 30 19:29:35.895330 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.895147    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:29:35.895330 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.895277    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:29:35.895818 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.895341    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:29:35.897871 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.897853    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kspfd"
Oct 30 19:29:35.899160 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.899143    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:35.900943 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.900353    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:35.900943 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:35.900414    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:29:35.900943 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.900419    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 19:29:35.901981 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.901609    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:35.911404 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911388    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-sys\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:35.911492 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6v4c\" (UniqueName: \"kubernetes.io/projected/e0a98773-9cb1-4637-9188-52094d47dceb-kube-api-access-d6v4c\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:35.911552 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de8ddc3f-5d23-4cff-9f2e-9167700ff109-ovnkube-script-lib\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.911552 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d50ec904-b3c9-4677-9624-39cb202f65c9-cnibin\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:35.911648 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/23de5b1c-4825-43ed-bff8-cdab8789989e-agent-certs\") pod \"konnectivity-agent-kspfd\" (UID: \"23de5b1c-4825-43ed-bff8-cdab8789989e\") " pod="kube-system/konnectivity-agent-kspfd"
Oct 30 19:29:35.911648 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-modprobe-d\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:35.911747 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-systemd\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:35.911747 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911674    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-systemd-units\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.911747 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-cnibin\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:35.911889 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911749    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-run-openvswitch\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.911889 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911772    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-node-log\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.911889 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911795    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-log-socket\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.911889 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-cni-bin\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.911889 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:35.912140 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-sys-fs\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:35.912140 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53bd98ea-ea87-484a-b7b1-8c9070fc9b25-serviceca\") pod \"node-ca-h5m52\" (UID: \"53bd98ea-ea87-484a-b7b1-8c9070fc9b25\") " pod="openshift-image-registry/node-ca-h5m52"
Oct 30 19:29:35.912140 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911954    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-lib-modules\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:35.912140 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e0a98773-9cb1-4637-9188-52094d47dceb-etc-tuned\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:35.912140 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.911997    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-slash\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.912140 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6hdz\" (UniqueName: \"kubernetes.io/projected/de8ddc3f-5d23-4cff-9f2e-9167700ff109-kube-api-access-f6hdz\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.912140 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-cni-binary-copy\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:35.912140 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-socket-dir-parent\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:35.912140 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-run-k8s-cni-cncf-io\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:35.912534 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-sysctl-d\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:35.912534 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912226    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.912534 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de8ddc3f-5d23-4cff-9f2e-9167700ff109-ovn-node-metrics-cert\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.912534 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tcgh\" (UniqueName: \"kubernetes.io/projected/68ff550a-daae-421b-9dd6-e9b1f33585d1-kube-api-access-9tcgh\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:35.912534 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912369    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-cni-dir\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:35.912534 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912409    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs\") pod \"network-metrics-daemon-mwcmw\" (UID: \"7c979e36-8fff-4401-a991-1a661fd9c6e5\") " pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:35.912534 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d50ec904-b3c9-4677-9624-39cb202f65c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:35.912534 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d50ec904-b3c9-4677-9624-39cb202f65c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:35.912534 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912495    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-system-cni-dir\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:35.912835 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912548    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-daemon-config\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:35.912835 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-run-systemd\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.912835 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-run-ovn\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.912835 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de8ddc3f-5d23-4cff-9f2e-9167700ff109-env-overrides\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.912835 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmjjp\" (UniqueName: \"kubernetes.io/projected/d50ec904-b3c9-4677-9624-39cb202f65c9-kube-api-access-pmjjp\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:35.912835 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6scd\" (UniqueName: \"kubernetes.io/projected/e2ac53be-96b4-4976-9581-b38cc97e1eb0-kube-api-access-l6scd\") pod \"iptables-alerter-5n649\" (UID: \"e2ac53be-96b4-4976-9581-b38cc97e1eb0\") " pod="openshift-network-operator/iptables-alerter-5n649"
Oct 30 19:29:35.912835 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-socket-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:35.912835 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8013e357-0fa2-4360-924b-18cbfe588b5c-kubelet-config\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:35.912835 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-var-lib-openvswitch\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.912835 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2836676-0428-4f2f-a801-0a25df87aee2-tmp-dir\") pod \"node-resolver-gmrbv\" (UID: \"c2836676-0428-4f2f-a801-0a25df87aee2\") " pod="openshift-dns/node-resolver-gmrbv"
Oct 30 19:29:35.912835 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-run-netns\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:35.912835 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e2ac53be-96b4-4976-9581-b38cc97e1eb0-iptables-alerter-script\") pod \"iptables-alerter-5n649\" (UID: \"e2ac53be-96b4-4976-9581-b38cc97e1eb0\") " pod="openshift-network-operator/iptables-alerter-5n649"
Oct 30 19:29:35.913262 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-var-lib-cni-bin\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:35.913262 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.912873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0a98773-9cb1-4637-9188-52094d47dceb-tmp\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:35.913262 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913003    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2ac53be-96b4-4976-9581-b38cc97e1eb0-host-slash\") pod \"iptables-alerter-5n649\" (UID: \"e2ac53be-96b4-4976-9581-b38cc97e1eb0\") " pod="openshift-network-operator/iptables-alerter-5n649"
Oct 30 19:29:35.913262 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-device-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:35.913262 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:35.913262 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-var-lib-cni-multus\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:35.913262 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-host\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:35.913262 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d50ec904-b3c9-4677-9624-39cb202f65c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:35.913262 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913232    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8013e357-0fa2-4360-924b-18cbfe588b5c-dbus\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:35.913692 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/23de5b1c-4825-43ed-bff8-cdab8789989e-konnectivity-ca\") pod \"konnectivity-agent-kspfd\" (UID: \"23de5b1c-4825-43ed-bff8-cdab8789989e\") " pod="kube-system/konnectivity-agent-kspfd"
Oct 30 19:29:35.913692 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-run-multus-certs\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:35.913692 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-etc-openvswitch\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.913692 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-run-ovn-kubernetes\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.913692 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de8ddc3f-5d23-4cff-9f2e-9167700ff109-ovnkube-config\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.913692 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913399    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53bd98ea-ea87-484a-b7b1-8c9070fc9b25-host\") pod \"node-ca-h5m52\" (UID: \"53bd98ea-ea87-484a-b7b1-8c9070fc9b25\") " pod="openshift-image-registry/node-ca-h5m52"
Oct 30 19:29:35.913692 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-sysctl-conf\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:35.913692 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913446    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-var-lib-kubelet\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:35.913692 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csj5l\" (UniqueName: \"kubernetes.io/projected/7c979e36-8fff-4401-a991-1a661fd9c6e5-kube-api-access-csj5l\") pod \"network-metrics-daemon-mwcmw\" (UID: \"7c979e36-8fff-4401-a991-1a661fd9c6e5\") " pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:35.913692 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-var-lib-kubelet\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:35.913692 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-hostroot\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:35.913692 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c2836676-0428-4f2f-a801-0a25df87aee2-hosts-file\") pod \"node-resolver-gmrbv\" (UID: \"c2836676-0428-4f2f-a801-0a25df87aee2\") " pod="openshift-dns/node-resolver-gmrbv"
Oct 30 19:29:35.913692 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9tds\" (UniqueName: \"kubernetes.io/projected/53bd98ea-ea87-484a-b7b1-8c9070fc9b25-kube-api-access-b9tds\") pod \"node-ca-h5m52\" (UID: \"53bd98ea-ea87-484a-b7b1-8c9070fc9b25\") " pod="openshift-image-registry/node-ca-h5m52"
Oct 30 19:29:35.913692 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-sysconfig\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:35.913692 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913632    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-run\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:35.913692 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5z5s\" (UniqueName: \"kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s\") pod \"network-check-target-vwkv5\" (UID: \"101b32f8-ea50-4c3d-8114-15ebe26eb6d0\") " pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:35.914440 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-kubelet\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.914440 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-cni-netd\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.914440 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-kubernetes\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:35.914440 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913870    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d50ec904-b3c9-4677-9624-39cb202f65c9-system-cni-dir\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:35.914440 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d50ec904-b3c9-4677-9624-39cb202f65c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:35.914440 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-registration-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:35.914440 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-conf-dir\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:35.914440 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.913981    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-etc-kubernetes\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:35.914440 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.914006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9fpq\" (UniqueName: \"kubernetes.io/projected/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-kube-api-access-k9fpq\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:35.914440 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.914045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-run-netns\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:35.914440 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.914073    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbkb8\" (UniqueName: \"kubernetes.io/projected/c2836676-0428-4f2f-a801-0a25df87aee2-kube-api-access-xbkb8\") pod \"node-resolver-gmrbv\" (UID: \"c2836676-0428-4f2f-a801-0a25df87aee2\") " pod="openshift-dns/node-resolver-gmrbv"
Oct 30 19:29:35.914440 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.914097    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:35.914985 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.914695    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:29:35.956673 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.956650    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-84xsz"
Oct 30 19:29:35.968602 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.968584    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-84xsz"
Oct 30 19:29:35.990804 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.990777    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:29:35.992627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.991073    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:29:35.992627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.991286    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:29:35.992627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.991417    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:29:35.992627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.991448    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:29:35.992627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.991521    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:29:35.992627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.991654    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:29:35.992627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.991791    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:29:35.992627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.991811    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:29:35.992627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.991933    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:29:35.992627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.992072    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:29:35.992627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.992135    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:29:35.992627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.992199    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2cs46\""
Oct 30 19:29:35.992627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.992326    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:29:35.992627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.992356    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wp4kx\""
Oct 30 19:29:35.992627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.992500    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p6d49\""
Oct 30 19:29:35.992627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.992537    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z724k\""
Oct 30 19:29:35.992627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.992548    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:29:35.992627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.992569    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fbg5p\""
Oct 30 19:29:35.993455 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.992727    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:29:35.993455 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:35.992736    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:29:36.001881 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.001864    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:29:36.014638 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.014616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-cni-dir\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.014754 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.014663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs\") pod \"network-metrics-daemon-mwcmw\" (UID: \"7c979e36-8fff-4401-a991-1a661fd9c6e5\") " pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:36.014754 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.014691    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d50ec904-b3c9-4677-9624-39cb202f65c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:36.014754 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.014715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d50ec904-b3c9-4677-9624-39cb202f65c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:36.014754 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.014727    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-cni-dir\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.014933 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:36.014834    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:36.014933 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.014875    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-system-cni-dir\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.014933 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.014878    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d50ec904-b3c9-4677-9624-39cb202f65c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:36.014933 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.014830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-system-cni-dir\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.014933 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:36.014916    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs podName:7c979e36-8fff-4401-a991-1a661fd9c6e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:36.514880613 +0000 UTC m=+2.929519587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs") pod "network-metrics-daemon-mwcmw" (UID: "7c979e36-8fff-4401-a991-1a661fd9c6e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:36.015115 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.014993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-daemon-config\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.015115 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-run-systemd\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.015115 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-run-ovn\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.015115 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de8ddc3f-5d23-4cff-9f2e-9167700ff109-env-overrides\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.015115 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjjp\" (UniqueName: \"kubernetes.io/projected/d50ec904-b3c9-4677-9624-39cb202f65c9-kube-api-access-pmjjp\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:36.015115 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6scd\" (UniqueName: \"kubernetes.io/projected/e2ac53be-96b4-4976-9581-b38cc97e1eb0-kube-api-access-l6scd\") pod \"iptables-alerter-5n649\" (UID: \"e2ac53be-96b4-4976-9581-b38cc97e1eb0\") " pod="openshift-network-operator/iptables-alerter-5n649"
Oct 30 19:29:36.015294 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-socket-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:36.015294 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015132    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-run-ovn\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.015294 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8013e357-0fa2-4360-924b-18cbfe588b5c-kubelet-config\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:36.015294 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015174    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8013e357-0fa2-4360-924b-18cbfe588b5c-kubelet-config\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:36.015294 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-var-lib-openvswitch\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.015294 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015207    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2836676-0428-4f2f-a801-0a25df87aee2-tmp-dir\") pod \"node-resolver-gmrbv\" (UID: \"c2836676-0428-4f2f-a801-0a25df87aee2\") " pod="openshift-dns/node-resolver-gmrbv"
Oct 30 19:29:36.015294 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-run-netns\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.015294 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e2ac53be-96b4-4976-9581-b38cc97e1eb0-iptables-alerter-script\") pod \"iptables-alerter-5n649\" (UID: \"e2ac53be-96b4-4976-9581-b38cc97e1eb0\") " pod="openshift-network-operator/iptables-alerter-5n649"
Oct 30 19:29:36.015294 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015262    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d50ec904-b3c9-4677-9624-39cb202f65c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:36.015294 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-var-lib-cni-bin\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.015572 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015297    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0a98773-9cb1-4637-9188-52094d47dceb-tmp\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.015572 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015537    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de8ddc3f-5d23-4cff-9f2e-9167700ff109-env-overrides\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.015572 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015542    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:29:36.015572 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015556    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2836676-0428-4f2f-a801-0a25df87aee2-tmp-dir\") pod \"node-resolver-gmrbv\" (UID: \"c2836676-0428-4f2f-a801-0a25df87aee2\") " pod="openshift-dns/node-resolver-gmrbv"
Oct 30 19:29:36.015677 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2ac53be-96b4-4976-9581-b38cc97e1eb0-host-slash\") pod \"iptables-alerter-5n649\" (UID: \"e2ac53be-96b4-4976-9581-b38cc97e1eb0\") " pod="openshift-network-operator/iptables-alerter-5n649"
Oct 30 19:29:36.015677 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015626    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-socket-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:36.015677 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-device-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:36.015677 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:36.015677 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015665    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-run-systemd\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.015807 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-var-lib-cni-multus\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.015807 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015701    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-run-netns\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.015807 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015780    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2ac53be-96b4-4976-9581-b38cc97e1eb0-host-slash\") pod \"iptables-alerter-5n649\" (UID: \"e2ac53be-96b4-4976-9581-b38cc97e1eb0\") " pod="openshift-network-operator/iptables-alerter-5n649"
Oct 30 19:29:36.015807 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-host\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.015807 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015801    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-var-lib-cni-bin\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.016000 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015805    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-device-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:36.016000 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d50ec904-b3c9-4677-9624-39cb202f65c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:36.016000 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015833    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-var-lib-openvswitch\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.016000 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8013e357-0fa2-4360-924b-18cbfe588b5c-dbus\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:36.016000 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/23de5b1c-4825-43ed-bff8-cdab8789989e-konnectivity-ca\") pod \"konnectivity-agent-kspfd\" (UID: \"23de5b1c-4825-43ed-bff8-cdab8789989e\") " pod="kube-system/konnectivity-agent-kspfd"
Oct 30 19:29:36.016000 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015874    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-var-lib-cni-multus\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.016000 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015900    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-host\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.016000 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015958    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:36.016000 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.015992    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8013e357-0fa2-4360-924b-18cbfe588b5c-dbus\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:36.016361 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-run-multus-certs\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.016361 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-etc-openvswitch\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.016361 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016070    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-run-ovn-kubernetes\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.016361 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016084    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-run-multus-certs\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.016361 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de8ddc3f-5d23-4cff-9f2e-9167700ff109-ovnkube-config\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.016361 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016127    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-etc-openvswitch\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.016361 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53bd98ea-ea87-484a-b7b1-8c9070fc9b25-host\") pod \"node-ca-h5m52\" (UID: \"53bd98ea-ea87-484a-b7b1-8c9070fc9b25\") " pod="openshift-image-registry/node-ca-h5m52"
Oct 30 19:29:36.016361 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016176    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-sysctl-conf\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.016361 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-var-lib-kubelet\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.016361 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csj5l\" (UniqueName: \"kubernetes.io/projected/7c979e36-8fff-4401-a991-1a661fd9c6e5-kube-api-access-csj5l\") pod \"network-metrics-daemon-mwcmw\" (UID: \"7c979e36-8fff-4401-a991-1a661fd9c6e5\") " pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:36.016361 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-var-lib-kubelet\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.016361 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016265    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-daemon-config\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.016361 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-hostroot\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.016361 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016297    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-hostroot\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.016361 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c2836676-0428-4f2f-a801-0a25df87aee2-hosts-file\") pod \"node-resolver-gmrbv\" (UID: \"c2836676-0428-4f2f-a801-0a25df87aee2\") " pod="openshift-dns/node-resolver-gmrbv"
Oct 30 19:29:36.016361 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9tds\" (UniqueName: \"kubernetes.io/projected/53bd98ea-ea87-484a-b7b1-8c9070fc9b25-kube-api-access-b9tds\") pod \"node-ca-h5m52\" (UID: \"53bd98ea-ea87-484a-b7b1-8c9070fc9b25\") " pod="openshift-image-registry/node-ca-h5m52"
Oct 30 19:29:36.016361 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-sysconfig\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.017011 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016372    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e2ac53be-96b4-4976-9581-b38cc97e1eb0-iptables-alerter-script\") pod \"iptables-alerter-5n649\" (UID: \"e2ac53be-96b4-4976-9581-b38cc97e1eb0\") " pod="openshift-network-operator/iptables-alerter-5n649"
Oct 30 19:29:36.017011 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016379    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-run\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.017011 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5z5s\" (UniqueName: \"kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s\") pod \"network-check-target-vwkv5\" (UID: \"101b32f8-ea50-4c3d-8114-15ebe26eb6d0\") " pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:36.017011 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-kubelet\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.017011 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-cni-netd\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.017011 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016468    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-sysctl-conf\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.017011 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016472    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/23de5b1c-4825-43ed-bff8-cdab8789989e-konnectivity-ca\") pod \"konnectivity-agent-kspfd\" (UID: \"23de5b1c-4825-43ed-bff8-cdab8789989e\") " pod="kube-system/konnectivity-agent-kspfd"
Oct 30 19:29:36.017011 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016472    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d50ec904-b3c9-4677-9624-39cb202f65c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:36.017011 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016335    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53bd98ea-ea87-484a-b7b1-8c9070fc9b25-host\") pod \"node-ca-h5m52\" (UID: \"53bd98ea-ea87-484a-b7b1-8c9070fc9b25\") " pod="openshift-image-registry/node-ca-h5m52"
Oct 30 19:29:36.017011 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-kubernetes\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.017011 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d50ec904-b3c9-4677-9624-39cb202f65c9-system-cni-dir\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:36.017011 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d50ec904-b3c9-4677-9624-39cb202f65c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:36.017011 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-registration-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:36.017011 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016573    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-var-lib-kubelet\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.017011 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016609    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c2836676-0428-4f2f-a801-0a25df87aee2-hosts-file\") pod \"node-resolver-gmrbv\" (UID: \"c2836676-0428-4f2f-a801-0a25df87aee2\") " pod="openshift-dns/node-resolver-gmrbv"
Oct 30 19:29:36.017011 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016622    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-var-lib-kubelet\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.017011 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016660    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-registration-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:36.018005 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016693    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-kubelet\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.018005 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016700    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-run-ovn-kubernetes\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.018005 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016711    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d50ec904-b3c9-4677-9624-39cb202f65c9-system-cni-dir\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:36.018005 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016734    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-kubernetes\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.018005 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016740    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-cni-netd\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.018005 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-conf-dir\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.018005 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016760    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-sysconfig\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.018005 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-etc-kubernetes\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.018005 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016790    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-conf-dir\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.018005 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016811    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-run\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.018005 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9fpq\" (UniqueName: \"kubernetes.io/projected/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-kube-api-access-k9fpq\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.018005 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-run-netns\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.018005 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbkb8\" (UniqueName: \"kubernetes.io/projected/c2836676-0428-4f2f-a801-0a25df87aee2-kube-api-access-xbkb8\") pod \"node-resolver-gmrbv\" (UID: \"c2836676-0428-4f2f-a801-0a25df87aee2\") " pod="openshift-dns/node-resolver-gmrbv"
Oct 30 19:29:36.018005 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016894    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-etc-kubernetes\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.018005 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016915    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-run-netns\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.018005 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.016977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:36.018005 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:36.017085    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:36.018005 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-sys\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.018819 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:36.017162    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret podName:8013e357-0fa2-4360-924b-18cbfe588b5c nodeName:}" failed. No retries permitted until 2026-10-30 19:29:36.517145783 +0000 UTC m=+2.931784757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret") pod "global-pull-secret-syncer-d2hv2" (UID: "8013e357-0fa2-4360-924b-18cbfe588b5c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:36.018819 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017182    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6v4c\" (UniqueName: \"kubernetes.io/projected/e0a98773-9cb1-4637-9188-52094d47dceb-kube-api-access-d6v4c\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.018819 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017192    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-sys\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.018819 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de8ddc3f-5d23-4cff-9f2e-9167700ff109-ovnkube-script-lib\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.018819 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d50ec904-b3c9-4677-9624-39cb202f65c9-cnibin\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:36.018819 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/23de5b1c-4825-43ed-bff8-cdab8789989e-agent-certs\") pod \"konnectivity-agent-kspfd\" (UID: \"23de5b1c-4825-43ed-bff8-cdab8789989e\") " pod="kube-system/konnectivity-agent-kspfd"
Oct 30 19:29:36.018819 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-modprobe-d\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.018819 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-systemd\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.018819 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017523    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-modprobe-d\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.018819 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-systemd-units\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.018819 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017544    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de8ddc3f-5d23-4cff-9f2e-9167700ff109-ovnkube-config\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.018819 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017584    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d50ec904-b3c9-4677-9624-39cb202f65c9-cnibin\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:36.018819 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017600    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-systemd\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.018819 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-cnibin\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.018819 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017623    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-systemd-units\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.018819 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-run-openvswitch\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.018819 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017670    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-cnibin\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.019560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-node-log\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.019560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017703    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-run-openvswitch\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.019560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017727    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-log-socket\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.019560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-cni-bin\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.019560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017773    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-log-socket\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.019560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017775    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-node-log\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.019560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:36.019560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017802    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-cni-bin\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.019560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-sys-fs\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:36.019560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017822    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:36.019560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53bd98ea-ea87-484a-b7b1-8c9070fc9b25-serviceca\") pod \"node-ca-h5m52\" (UID: \"53bd98ea-ea87-484a-b7b1-8c9070fc9b25\") " pod="openshift-image-registry/node-ca-h5m52"
Oct 30 19:29:36.019560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017854    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-sys-fs\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:36.019560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-lib-modules\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.019560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e0a98773-9cb1-4637-9188-52094d47dceb-etc-tuned\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.019560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-slash\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.019560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6hdz\" (UniqueName: \"kubernetes.io/projected/de8ddc3f-5d23-4cff-9f2e-9167700ff109-kube-api-access-f6hdz\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.019560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-cni-binary-copy\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.019560 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017949    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-lib-modules\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.020301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.017987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-socket-dir-parent\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.020301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.018012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-run-k8s-cni-cncf-io\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.020301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.018052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-sysctl-d\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.020301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.018078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.020301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.018105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de8ddc3f-5d23-4cff-9f2e-9167700ff109-ovn-node-metrics-cert\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.020301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.018130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tcgh\" (UniqueName: \"kubernetes.io/projected/68ff550a-daae-421b-9dd6-e9b1f33585d1-kube-api-access-9tcgh\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:36.020301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.018211    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-slash\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.020301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.018236    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-run-k8s-cni-cncf-io\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.020301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.018268    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.020301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.018276    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de8ddc3f-5d23-4cff-9f2e-9167700ff109-ovnkube-script-lib\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.020301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.018288    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53bd98ea-ea87-484a-b7b1-8c9070fc9b25-serviceca\") pod \"node-ca-h5m52\" (UID: \"53bd98ea-ea87-484a-b7b1-8c9070fc9b25\") " pod="openshift-image-registry/node-ca-h5m52"
Oct 30 19:29:36.020301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.018355    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-socket-dir-parent\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.020301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.018366    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-sysctl-d\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.020301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.018478    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d50ec904-b3c9-4677-9624-39cb202f65c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:36.020301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.018742    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-cni-binary-copy\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.020301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.020124    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0a98773-9cb1-4637-9188-52094d47dceb-tmp\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.020301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.020151    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e0a98773-9cb1-4637-9188-52094d47dceb-etc-tuned\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.021108 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.021066    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de8ddc3f-5d23-4cff-9f2e-9167700ff109-ovn-node-metrics-cert\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.021108 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.021080    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/23de5b1c-4825-43ed-bff8-cdab8789989e-agent-certs\") pod \"konnectivity-agent-kspfd\" (UID: \"23de5b1c-4825-43ed-bff8-cdab8789989e\") " pod="kube-system/konnectivity-agent-kspfd"
Oct 30 19:29:36.026398 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:36.026378    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:29:36.026478 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:36.026402    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:29:36.026478 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:36.026412    2577 projected.go:196] Error preparing data for projected volume kube-api-access-d5z5s for pod openshift-network-diagnostics/network-check-target-vwkv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:36.026478 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:36.026450    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s podName:101b32f8-ea50-4c3d-8114-15ebe26eb6d0 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:36.52643919 +0000 UTC m=+2.941078165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d5z5s" (UniqueName: "kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s") pod "network-check-target-vwkv5" (UID: "101b32f8-ea50-4c3d-8114-15ebe26eb6d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:36.029377 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.029353    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csj5l\" (UniqueName: \"kubernetes.io/projected/7c979e36-8fff-4401-a991-1a661fd9c6e5-kube-api-access-csj5l\") pod \"network-metrics-daemon-mwcmw\" (UID: \"7c979e36-8fff-4401-a991-1a661fd9c6e5\") " pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:36.029487 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.029467    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbkb8\" (UniqueName: \"kubernetes.io/projected/c2836676-0428-4f2f-a801-0a25df87aee2-kube-api-access-xbkb8\") pod \"node-resolver-gmrbv\" (UID: \"c2836676-0428-4f2f-a801-0a25df87aee2\") " pod="openshift-dns/node-resolver-gmrbv"
Oct 30 19:29:36.029936 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.029916    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjjp\" (UniqueName: \"kubernetes.io/projected/d50ec904-b3c9-4677-9624-39cb202f65c9-kube-api-access-pmjjp\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:36.030316 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.030298    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9fpq\" (UniqueName: \"kubernetes.io/projected/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-kube-api-access-k9fpq\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.030496 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.030471    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6scd\" (UniqueName: \"kubernetes.io/projected/e2ac53be-96b4-4976-9581-b38cc97e1eb0-kube-api-access-l6scd\") pod \"iptables-alerter-5n649\" (UID: \"e2ac53be-96b4-4976-9581-b38cc97e1eb0\") " pod="openshift-network-operator/iptables-alerter-5n649"
Oct 30 19:29:36.030838 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.030822    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9tds\" (UniqueName: \"kubernetes.io/projected/53bd98ea-ea87-484a-b7b1-8c9070fc9b25-kube-api-access-b9tds\") pod \"node-ca-h5m52\" (UID: \"53bd98ea-ea87-484a-b7b1-8c9070fc9b25\") " pod="openshift-image-registry/node-ca-h5m52"
Oct 30 19:29:36.034603 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.034582    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6hdz\" (UniqueName: \"kubernetes.io/projected/de8ddc3f-5d23-4cff-9f2e-9167700ff109-kube-api-access-f6hdz\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.035089 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.035069    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6v4c\" (UniqueName: \"kubernetes.io/projected/e0a98773-9cb1-4637-9188-52094d47dceb-kube-api-access-d6v4c\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.036251 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.036232    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tcgh\" (UniqueName: \"kubernetes.io/projected/68ff550a-daae-421b-9dd6-e9b1f33585d1-kube-api-access-9tcgh\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:36.110953 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.110889    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal" event={"ID":"d4d61e23ca703103c56d5c7a000f30c8","Type":"ContainerStarted","Data":"c3e72d26b223d07b1cab9286bc95e64e8f5540868af1598407589ef50ee834aa"}
Oct 30 19:29:36.112164 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.112134    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal" event={"ID":"7dbf81fa0f84f4c1de7386fb2838c009","Type":"ContainerStarted","Data":"f5b879a0b0afdc510709a56fedddbe3e63eee4eb9a2d869df77a7d9f3235cde6"}
Oct 30 19:29:36.198606 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.198529    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h5m52"
Oct 30 19:29:36.206838 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.206818    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5n649"
Oct 30 19:29:36.209019 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:36.208988    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bd98ea_ea87_484a_b7b1_8c9070fc9b25.slice/crio-90341f35811b436dab6f5eeb57758a8f6c6c901b850325b0566b18661bd1a6ed WatchSource:0}: Error finding container 90341f35811b436dab6f5eeb57758a8f6c6c901b850325b0566b18661bd1a6ed: Status 404 returned error can't find the container with id 90341f35811b436dab6f5eeb57758a8f6c6c901b850325b0566b18661bd1a6ed
Oct 30 19:29:36.214552 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:36.214512    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ac53be_96b4_4976_9581_b38cc97e1eb0.slice/crio-905a4f91743ddd1fb3ee8d85a3d61109230e811ed1b2c825e5e05dae8ccd8b7b WatchSource:0}: Error finding container 905a4f91743ddd1fb3ee8d85a3d61109230e811ed1b2c825e5e05dae8ccd8b7b: Status 404 returned error can't find the container with id 905a4f91743ddd1fb3ee8d85a3d61109230e811ed1b2c825e5e05dae8ccd8b7b
Oct 30 19:29:36.215174 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.214999    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:29:36.223700 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.223679    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gmrbv"
Oct 30 19:29:36.223793 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:36.223775    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a98773_9cb1_4637_9188_52094d47dceb.slice/crio-5ee14fddfa7ca96c50f4607a2a11010fb5f59ffe51405f61583e827e9dea80c7 WatchSource:0}: Error finding container 5ee14fddfa7ca96c50f4607a2a11010fb5f59ffe51405f61583e827e9dea80c7: Status 404 returned error can't find the container with id 5ee14fddfa7ca96c50f4607a2a11010fb5f59ffe51405f61583e827e9dea80c7
Oct 30 19:29:36.232336 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:36.232315    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2836676_0428_4f2f_a801_0a25df87aee2.slice/crio-cfd93681f5f218e22c883a9e49151eb7f586674f01d276757174f9829a421c37 WatchSource:0}: Error finding container cfd93681f5f218e22c883a9e49151eb7f586674f01d276757174f9829a421c37: Status 404 returned error can't find the container with id cfd93681f5f218e22c883a9e49151eb7f586674f01d276757174f9829a421c37
Oct 30 19:29:36.233049 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.232889    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:29:36.246667 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.246538    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:36.247645 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.247624    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kspfd"
Oct 30 19:29:36.256438 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:36.256413    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde8ddc3f_5d23_4cff_9f2e_9167700ff109.slice/crio-c6385dafaa9f8c8e977f2d44f2491d84e9755c9c1ae80b00e04f2ca29bcdbf8b WatchSource:0}: Error finding container c6385dafaa9f8c8e977f2d44f2491d84e9755c9c1ae80b00e04f2ca29bcdbf8b: Status 404 returned error can't find the container with id c6385dafaa9f8c8e977f2d44f2491d84e9755c9c1ae80b00e04f2ca29bcdbf8b
Oct 30 19:29:36.258497 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:36.258477    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23de5b1c_4825_43ed_bff8_cdab8789989e.slice/crio-ed3d450f2f9f551f3fe9791bd70422e1170a01b7f13bf5a069d6f897d8939011 WatchSource:0}: Error finding container ed3d450f2f9f551f3fe9791bd70422e1170a01b7f13bf5a069d6f897d8939011: Status 404 returned error can't find the container with id ed3d450f2f9f551f3fe9791bd70422e1170a01b7f13bf5a069d6f897d8939011
Oct 30 19:29:36.259867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.259804    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:29:36.260755 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.260739    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d6gqm"
Oct 30 19:29:36.268083 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:36.268066    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50ec904_b3c9_4677_9624_39cb202f65c9.slice/crio-c40aa314795318f3f4dd9529bcfa37edcc6c76c12978d657b607d6cd3ae69209 WatchSource:0}: Error finding container c40aa314795318f3f4dd9529bcfa37edcc6c76c12978d657b607d6cd3ae69209: Status 404 returned error can't find the container with id c40aa314795318f3f4dd9529bcfa37edcc6c76c12978d657b607d6cd3ae69209
Oct 30 19:29:36.273249 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:29:36.273228    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df58e7f_8f5f_4e3a_af1d_7e29e4b58c57.slice/crio-f4174a924aa6c72d9f7eeab913165b762220275baf3d845126dc6629a256070b WatchSource:0}: Error finding container f4174a924aa6c72d9f7eeab913165b762220275baf3d845126dc6629a256070b: Status 404 returned error can't find the container with id f4174a924aa6c72d9f7eeab913165b762220275baf3d845126dc6629a256070b
Oct 30 19:29:36.278663 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.278623    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:29:36.520663 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.520638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:36.520782 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.520693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs\") pod \"network-metrics-daemon-mwcmw\" (UID: \"7c979e36-8fff-4401-a991-1a661fd9c6e5\") " pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:36.520844 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:36.520799    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:36.520882 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:36.520868    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret podName:8013e357-0fa2-4360-924b-18cbfe588b5c nodeName:}" failed. No retries permitted until 2026-10-30 19:29:37.520850401 +0000 UTC m=+3.935489381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret") pod "global-pull-secret-syncer-d2hv2" (UID: "8013e357-0fa2-4360-924b-18cbfe588b5c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:36.520938 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:36.520802    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:36.520982 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:36.520947    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs podName:7c979e36-8fff-4401-a991-1a661fd9c6e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:37.520930469 +0000 UTC m=+3.935569437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs") pod "network-metrics-daemon-mwcmw" (UID: "7c979e36-8fff-4401-a991-1a661fd9c6e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:36.621880 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.621857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5z5s\" (UniqueName: \"kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s\") pod \"network-check-target-vwkv5\" (UID: \"101b32f8-ea50-4c3d-8114-15ebe26eb6d0\") " pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:36.622008 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:36.621988    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:29:36.622077 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:36.622017    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:29:36.622077 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:36.622044    2577 projected.go:196] Error preparing data for projected volume kube-api-access-d5z5s for pod openshift-network-diagnostics/network-check-target-vwkv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:36.622163 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:36.622106    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s podName:101b32f8-ea50-4c3d-8114-15ebe26eb6d0 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:37.622084858 +0000 UTC m=+4.036723833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5z5s" (UniqueName: "kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s") pod "network-check-target-vwkv5" (UID: "101b32f8-ea50-4c3d-8114-15ebe26eb6d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:36.970821 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.970484    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:24:35 +0000 UTC" deadline="2028-07-30 07:05:05.271034806 +0000 UTC"
Oct 30 19:29:36.970821 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:36.970756    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15323h35m28.300284197s"
Oct 30 19:29:37.107206 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:37.107064    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:37.107353 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:37.107244    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:29:37.107741 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:37.107583    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:37.107741 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:37.107683    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:29:37.125067 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:37.124494    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal" event={"ID":"d4d61e23ca703103c56d5c7a000f30c8","Type":"ContainerStarted","Data":"32bdffa2472c4e691d97a404f2c6ef5b915b01d0306683e281ad26c4e97ae4da"}
Oct 30 19:29:37.133787 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:37.133718    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kspfd" event={"ID":"23de5b1c-4825-43ed-bff8-cdab8789989e","Type":"ContainerStarted","Data":"ed3d450f2f9f551f3fe9791bd70422e1170a01b7f13bf5a069d6f897d8939011"}
Oct 30 19:29:37.140973 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:37.140876    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn" event={"ID":"68ff550a-daae-421b-9dd6-e9b1f33585d1","Type":"ContainerStarted","Data":"e30bb07f1caa152986c84a26125c0e5346b0e8bc552b5156f435783366de1ce9"}
Oct 30 19:29:37.155784 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:37.155759    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gmrbv" event={"ID":"c2836676-0428-4f2f-a801-0a25df87aee2","Type":"ContainerStarted","Data":"cfd93681f5f218e22c883a9e49151eb7f586674f01d276757174f9829a421c37"}
Oct 30 19:29:37.160870 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:37.160843    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h5m52" event={"ID":"53bd98ea-ea87-484a-b7b1-8c9070fc9b25","Type":"ContainerStarted","Data":"90341f35811b436dab6f5eeb57758a8f6c6c901b850325b0566b18661bd1a6ed"}
Oct 30 19:29:37.170418 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:37.169780    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal" event={"ID":"7dbf81fa0f84f4c1de7386fb2838c009","Type":"ContainerStarted","Data":"3c1e0992e64794dda6a03b52961bbbeea8b8c57dba17fc3bee4785e3d301d968"}
Oct 30 19:29:37.173234 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:37.173178    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d6gqm" event={"ID":"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57","Type":"ContainerStarted","Data":"f4174a924aa6c72d9f7eeab913165b762220275baf3d845126dc6629a256070b"}
Oct 30 19:29:37.175223 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:37.175142    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qmqjk" event={"ID":"d50ec904-b3c9-4677-9624-39cb202f65c9","Type":"ContainerStarted","Data":"c40aa314795318f3f4dd9529bcfa37edcc6c76c12978d657b607d6cd3ae69209"}
Oct 30 19:29:37.203050 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:37.199972    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr" event={"ID":"de8ddc3f-5d23-4cff-9f2e-9167700ff109","Type":"ContainerStarted","Data":"c6385dafaa9f8c8e977f2d44f2491d84e9755c9c1ae80b00e04f2ca29bcdbf8b"}
Oct 30 19:29:37.204269 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:37.204215    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bkdq5" event={"ID":"e0a98773-9cb1-4637-9188-52094d47dceb","Type":"ContainerStarted","Data":"5ee14fddfa7ca96c50f4607a2a11010fb5f59ffe51405f61583e827e9dea80c7"}
Oct 30 19:29:37.209107 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:37.205847    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5n649" event={"ID":"e2ac53be-96b4-4976-9581-b38cc97e1eb0","Type":"ContainerStarted","Data":"905a4f91743ddd1fb3ee8d85a3d61109230e811ed1b2c825e5e05dae8ccd8b7b"}
Oct 30 19:29:37.530832 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:37.530750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:37.530832 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:37.530812    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs\") pod \"network-metrics-daemon-mwcmw\" (UID: \"7c979e36-8fff-4401-a991-1a661fd9c6e5\") " pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:37.531059 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:37.530937    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:37.531059 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:37.530991    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs podName:7c979e36-8fff-4401-a991-1a661fd9c6e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:39.530975077 +0000 UTC m=+5.945614050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs") pod "network-metrics-daemon-mwcmw" (UID: "7c979e36-8fff-4401-a991-1a661fd9c6e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:37.531384 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:37.531365    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:37.531458 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:37.531420    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret podName:8013e357-0fa2-4360-924b-18cbfe588b5c nodeName:}" failed. No retries permitted until 2026-10-30 19:29:39.531406904 +0000 UTC m=+5.946045878 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret") pod "global-pull-secret-syncer-d2hv2" (UID: "8013e357-0fa2-4360-924b-18cbfe588b5c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:37.631802 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:37.631768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5z5s\" (UniqueName: \"kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s\") pod \"network-check-target-vwkv5\" (UID: \"101b32f8-ea50-4c3d-8114-15ebe26eb6d0\") " pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:37.631987 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:37.631971    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:29:37.632070 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:37.631996    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:29:37.632070 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:37.632008    2577 projected.go:196] Error preparing data for projected volume kube-api-access-d5z5s for pod openshift-network-diagnostics/network-check-target-vwkv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:37.632176 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:37.632073    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s podName:101b32f8-ea50-4c3d-8114-15ebe26eb6d0 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:39.632056382 +0000 UTC m=+6.046695355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5z5s" (UniqueName: "kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s") pod "network-check-target-vwkv5" (UID: "101b32f8-ea50-4c3d-8114-15ebe26eb6d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:37.971925 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:37.971840    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:24:35 +0000 UTC" deadline="2028-04-16 02:59:26.510250264 +0000 UTC"
Oct 30 19:29:37.971925 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:37.971885    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12799h29m48.538369979s"
Oct 30 19:29:38.119925 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:38.119453    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:38.119925 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:38.119585    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:29:38.229201 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:38.229119    2577 generic.go:356] "Generic (PLEG): container finished" podID="d4d61e23ca703103c56d5c7a000f30c8" containerID="32bdffa2472c4e691d97a404f2c6ef5b915b01d0306683e281ad26c4e97ae4da" exitCode=0
Oct 30 19:29:38.230158 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:38.230131    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal" event={"ID":"d4d61e23ca703103c56d5c7a000f30c8","Type":"ContainerDied","Data":"32bdffa2472c4e691d97a404f2c6ef5b915b01d0306683e281ad26c4e97ae4da"}
Oct 30 19:29:38.249273 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:38.249205    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal" podStartSLOduration=3.249190572 podStartE2EDuration="3.249190572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:37.188289691 +0000 UTC m=+3.602928675" watchObservedRunningTime="2026-10-30 19:29:38.249190572 +0000 UTC m=+4.663829553"
Oct 30 19:29:39.107784 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:39.107702    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:39.108262 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:39.107829    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:29:39.108262 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:39.108227    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:39.108417 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:39.108392    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:29:39.240585 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:39.240098    2577 generic.go:356] "Generic (PLEG): container finished" podID="d50ec904-b3c9-4677-9624-39cb202f65c9" containerID="8c64d40d5554ebffab25ca34a05655a9ee4aaa9547400bd1335e98cfac502ff9" exitCode=0
Oct 30 19:29:39.240585 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:39.240299    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qmqjk" event={"ID":"d50ec904-b3c9-4677-9624-39cb202f65c9","Type":"ContainerDied","Data":"8c64d40d5554ebffab25ca34a05655a9ee4aaa9547400bd1335e98cfac502ff9"}
Oct 30 19:29:39.245888 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:39.245854    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal" event={"ID":"d4d61e23ca703103c56d5c7a000f30c8","Type":"ContainerStarted","Data":"a0cab43135b6401705ca48d899f27ff1dd14bbded046cc530250836218fbd16e"}
Oct 30 19:29:39.288575 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:39.288524    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal" podStartSLOduration=4.288507629 podStartE2EDuration="4.288507629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:39.288010467 +0000 UTC m=+5.702649478" watchObservedRunningTime="2026-10-30 19:29:39.288507629 +0000 UTC m=+5.703146612"
Oct 30 19:29:39.548359 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:39.548282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:39.548359 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:39.548342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs\") pod \"network-metrics-daemon-mwcmw\" (UID: \"7c979e36-8fff-4401-a991-1a661fd9c6e5\") " pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:39.548561 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:39.548463    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:39.548561 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:39.548523    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs podName:7c979e36-8fff-4401-a991-1a661fd9c6e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:43.548506624 +0000 UTC m=+9.963145600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs") pod "network-metrics-daemon-mwcmw" (UID: "7c979e36-8fff-4401-a991-1a661fd9c6e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:39.548930 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:39.548905    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:39.549149 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:39.549126    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret podName:8013e357-0fa2-4360-924b-18cbfe588b5c nodeName:}" failed. No retries permitted until 2026-10-30 19:29:43.549109123 +0000 UTC m=+9.963748084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret") pod "global-pull-secret-syncer-d2hv2" (UID: "8013e357-0fa2-4360-924b-18cbfe588b5c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:39.649836 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:39.649167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5z5s\" (UniqueName: \"kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s\") pod \"network-check-target-vwkv5\" (UID: \"101b32f8-ea50-4c3d-8114-15ebe26eb6d0\") " pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:39.649836 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:39.649403    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:29:39.649836 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:39.649427    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:29:39.649836 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:39.649440    2577 projected.go:196] Error preparing data for projected volume kube-api-access-d5z5s for pod openshift-network-diagnostics/network-check-target-vwkv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:39.649836 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:39.649502    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s podName:101b32f8-ea50-4c3d-8114-15ebe26eb6d0 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:43.6494832 +0000 UTC m=+10.064122174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5z5s" (UniqueName: "kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s") pod "network-check-target-vwkv5" (UID: "101b32f8-ea50-4c3d-8114-15ebe26eb6d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:40.107165 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:40.107132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:40.107328 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:40.107277    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:29:40.256158 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:40.256021    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kspfd" event={"ID":"23de5b1c-4825-43ed-bff8-cdab8789989e","Type":"ContainerStarted","Data":"afb190f5e0d197131dc6ffc149d202c9555918fe14445edf6744e786cce04445"}
Oct 30 19:29:40.686601 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:40.686337    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kspfd"
Oct 30 19:29:40.687178 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:40.686944    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kspfd"
Oct 30 19:29:40.718822 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:40.718072    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kspfd" podStartSLOduration=2.9892929820000003 podStartE2EDuration="6.718056775s" totalImagesPullingTime="3.728763793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:40.275274784 +0000 UTC m=+6.689913757" watchObservedRunningTime="2026-10-30 19:29:40.718056775 +0000 UTC m=+7.132695762"
Oct 30 19:29:41.107901 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:41.107259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:41.107901 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:41.107393    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:29:41.107901 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:41.107772    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:41.107901 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:41.107853    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:29:42.108083 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:42.107250    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:42.108356 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:42.108200    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:29:42.265957 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:42.265886    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn" event={"ID":"68ff550a-daae-421b-9dd6-e9b1f33585d1","Type":"ContainerStarted","Data":"ea03fc0abb982e9e6e0ffb0eeee749b959973953b5da90e3196effa917632bc0"}
Oct 30 19:29:42.270408 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:42.270386    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:29:42.271195 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:42.271176    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h5m52" event={"ID":"53bd98ea-ea87-484a-b7b1-8c9070fc9b25","Type":"ContainerStarted","Data":"d0626236e7db345b96b9b993f36be971223c06a8a3df65d2a9328f2d985d8618"}
Oct 30 19:29:43.107535 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:43.106822    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:43.107535 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:43.106943    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:29:43.107535 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:43.107340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:43.107535 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:43.107448    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:29:43.580309 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:43.580280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs\") pod \"network-metrics-daemon-mwcmw\" (UID: \"7c979e36-8fff-4401-a991-1a661fd9c6e5\") " pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:43.580692 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:43.580354    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:43.580692 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:43.580478    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:43.580692 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:43.580540    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret podName:8013e357-0fa2-4360-924b-18cbfe588b5c nodeName:}" failed. No retries permitted until 2026-10-30 19:29:51.5805226 +0000 UTC m=+17.995161574 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret") pod "global-pull-secret-syncer-d2hv2" (UID: "8013e357-0fa2-4360-924b-18cbfe588b5c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:43.580943 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:43.580922    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:43.581053 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:43.580983    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs podName:7c979e36-8fff-4401-a991-1a661fd9c6e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:51.580965951 +0000 UTC m=+17.995604927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs") pod "network-metrics-daemon-mwcmw" (UID: "7c979e36-8fff-4401-a991-1a661fd9c6e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:43.639800 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:43.639778    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:29:43.681337 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:43.681308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5z5s\" (UniqueName: \"kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s\") pod \"network-check-target-vwkv5\" (UID: \"101b32f8-ea50-4c3d-8114-15ebe26eb6d0\") " pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:43.681510 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:43.681490    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:29:43.681612 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:43.681519    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:29:43.681612 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:43.681535    2577 projected.go:196] Error preparing data for projected volume kube-api-access-d5z5s for pod openshift-network-diagnostics/network-check-target-vwkv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:43.681612 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:43.681587    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s podName:101b32f8-ea50-4c3d-8114-15ebe26eb6d0 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:51.681568654 +0000 UTC m=+18.096207630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5z5s" (UniqueName: "kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s") pod "network-check-target-vwkv5" (UID: "101b32f8-ea50-4c3d-8114-15ebe26eb6d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:44.094821 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:44.094493    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:29:43.639796138Z","UUID":"b74970b8-cc89-469a-994a-b6dd12d5ecc6","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:29:44.096080 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:44.095916    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:29:44.096080 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:44.095946    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:29:44.110261 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:44.109903    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:44.110261 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:44.110004    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:29:44.282360 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:44.282264    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn" event={"ID":"68ff550a-daae-421b-9dd6-e9b1f33585d1","Type":"ContainerStarted","Data":"d0435d5161c4a932ea074963a8acce11b4cba610d7ce632a688e4d57ea75f91e"}
Oct 30 19:29:44.295062 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:44.295019    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d6gqm" event={"ID":"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57","Type":"ContainerStarted","Data":"71f902bf13462c7aff35aaedf69aaa4330eb488d9a78aca6148166880275c264"}
Oct 30 19:29:44.299596 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:44.299564    2577 generic.go:356] "Generic (PLEG): container finished" podID="d50ec904-b3c9-4677-9624-39cb202f65c9" containerID="326d0682233a45834c22a6f31b1f7c222c06a13c580555894723e16e649c9218" exitCode=0
Oct 30 19:29:44.299688 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:44.299605    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qmqjk" event={"ID":"d50ec904-b3c9-4677-9624-39cb202f65c9","Type":"ContainerDied","Data":"326d0682233a45834c22a6f31b1f7c222c06a13c580555894723e16e649c9218"}
Oct 30 19:29:44.317465 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:44.317421    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-d6gqm" podStartSLOduration=2.997906317 podStartE2EDuration="10.317404566s" totalImagesPullingTime="7.319498249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:44.317345543 +0000 UTC m=+10.731984526" watchObservedRunningTime="2026-10-30 19:29:44.317404566 +0000 UTC m=+10.732043551"
Oct 30 19:29:44.318312 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:44.318207    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h5m52" podStartSLOduration=5.421423011 podStartE2EDuration="10.318193518s" totalImagesPullingTime="4.896770507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:42.289790025 +0000 UTC m=+8.704428999" watchObservedRunningTime="2026-10-30 19:29:44.318193518 +0000 UTC m=+10.732832506"
Oct 30 19:29:45.107739 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:45.107710    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:45.108275 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:45.107922    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:29:45.108275 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:45.107973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:45.108275 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:45.108084    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:29:45.305353 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:45.304632    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn" event={"ID":"68ff550a-daae-421b-9dd6-e9b1f33585d1","Type":"ContainerStarted","Data":"7d2cbd7fbf4d4a695fa7c4d8c5d05b227857cc26d27b6d1774bfd305a3667c63"}
Oct 30 19:29:45.307488 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:45.307429    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gmrbv" event={"ID":"c2836676-0428-4f2f-a801-0a25df87aee2","Type":"ContainerStarted","Data":"9d4b75e5dfaa1768868b01aeba4e69d58668b72dfda5262f0c7db2419f7f0d6c"}
Oct 30 19:29:45.310596 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:45.310563    2577 generic.go:356] "Generic (PLEG): container finished" podID="d50ec904-b3c9-4677-9624-39cb202f65c9" containerID="df72d35bf995a8dca2dd05eb5df90c423caafc8099b0fc54bd34cc83a499bbdb" exitCode=0
Oct 30 19:29:45.310700 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:45.310660    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qmqjk" event={"ID":"d50ec904-b3c9-4677-9624-39cb202f65c9","Type":"ContainerDied","Data":"df72d35bf995a8dca2dd05eb5df90c423caafc8099b0fc54bd34cc83a499bbdb"}
Oct 30 19:29:45.326995 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:45.326955    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn" podStartSLOduration=3.145309936 podStartE2EDuration="11.326944732s" totalImagesPullingTime="8.181634796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:45.326544326 +0000 UTC m=+11.741183332" watchObservedRunningTime="2026-10-30 19:29:45.326944732 +0000 UTC m=+11.741583714"
Oct 30 19:29:45.348252 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:45.348213    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gmrbv" podStartSLOduration=3.380818767 podStartE2EDuration="11.348202735s" totalImagesPullingTime="7.967383968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:45.347700725 +0000 UTC m=+11.762339709" watchObservedRunningTime="2026-10-30 19:29:45.348202735 +0000 UTC m=+11.762841716"
Oct 30 19:29:46.107753 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:46.107722    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:46.108223 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:46.107838    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:29:46.315816 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:46.315783    2577 generic.go:356] "Generic (PLEG): container finished" podID="d50ec904-b3c9-4677-9624-39cb202f65c9" containerID="c3bfefde28bf8ece52936d9c16f70cb124846f9601604853ea603504807ada05" exitCode=0
Oct 30 19:29:46.315961 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:46.315865    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qmqjk" event={"ID":"d50ec904-b3c9-4677-9624-39cb202f65c9","Type":"ContainerDied","Data":"c3bfefde28bf8ece52936d9c16f70cb124846f9601604853ea603504807ada05"}
Oct 30 19:29:46.318248 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:46.318209    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bkdq5" event={"ID":"e0a98773-9cb1-4637-9188-52094d47dceb","Type":"ContainerStarted","Data":"9aec7846c499b1262e11c2604ef511df1de5f9a964b6792c7e3c8f367a22f56e"}
Oct 30 19:29:46.319721 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:46.319698    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5n649" event={"ID":"e2ac53be-96b4-4976-9581-b38cc97e1eb0","Type":"ContainerStarted","Data":"9a2ec2b5b9efdee5b0ef7992097aafafafd2e30ce56e83968e937097b0f28d6b"}
Oct 30 19:29:46.363346 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:46.363269    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5n649" podStartSLOduration=4.613972404 podStartE2EDuration="12.363253048s" totalImagesPullingTime="7.749280644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:46.363210634 +0000 UTC m=+12.777849640" watchObservedRunningTime="2026-10-30 19:29:46.363253048 +0000 UTC m=+12.777892031"
Oct 30 19:29:46.383721 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:46.383685    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bkdq5" podStartSLOduration=2.472535377 podStartE2EDuration="12.383674025s" totalImagesPullingTime="9.911138648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:46.38361993 +0000 UTC m=+12.798258912" watchObservedRunningTime="2026-10-30 19:29:46.383674025 +0000 UTC m=+12.798313006"
Oct 30 19:29:47.107605 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:47.107572    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:47.107765 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:47.107698    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:29:47.107821 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:47.107785    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:47.108447 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:47.107893    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:29:48.107415 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:48.107231    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:48.107562 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:48.107512    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:29:49.107475 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:49.107443    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:49.107897 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:49.107560    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:29:49.107897 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:49.107618    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:49.107897 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:49.107757    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:29:49.327941 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:49.327907    2577 generic.go:356] "Generic (PLEG): container finished" podID="d50ec904-b3c9-4677-9624-39cb202f65c9" containerID="ef7734784ff9c4000cd6d12427135f50dd1adf0a2d975ccd2021bac7da0768a4" exitCode=0
Oct 30 19:29:49.328096 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:49.327955    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qmqjk" event={"ID":"d50ec904-b3c9-4677-9624-39cb202f65c9","Type":"ContainerDied","Data":"ef7734784ff9c4000cd6d12427135f50dd1adf0a2d975ccd2021bac7da0768a4"}
Oct 30 19:29:50.107413 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:50.107382    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:50.107555 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:50.107504    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:29:50.333337 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:50.333274    2577 generic.go:356] "Generic (PLEG): container finished" podID="d50ec904-b3c9-4677-9624-39cb202f65c9" containerID="b820bd4d290ed7f6fc9913def85165541c71f2f0f5f42b907496d57b336b771d" exitCode=0
Oct 30 19:29:50.333337 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:50.333337    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qmqjk" event={"ID":"d50ec904-b3c9-4677-9624-39cb202f65c9","Type":"ContainerDied","Data":"b820bd4d290ed7f6fc9913def85165541c71f2f0f5f42b907496d57b336b771d"}
Oct 30 19:29:51.107505 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:51.107275    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:51.107681 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:51.107335    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:51.107681 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:51.107538    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:29:51.107681 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:51.107611    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:29:51.338869 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:51.338839    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qmqjk" event={"ID":"d50ec904-b3c9-4677-9624-39cb202f65c9","Type":"ContainerStarted","Data":"932c2cf28e2042dd0dc9fc5baaa93990b17ddf86e97c33f51aec1216843dfc66"}
Oct 30 19:29:51.341452 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:51.341428    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr" event={"ID":"de8ddc3f-5d23-4cff-9f2e-9167700ff109","Type":"ContainerStarted","Data":"e123e6a59cd4e968aa91e134f96f636cdee594a51ae51a02e4f2fad4db1001c1"}
Oct 30 19:29:51.341546 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:51.341490    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr" event={"ID":"de8ddc3f-5d23-4cff-9f2e-9167700ff109","Type":"ContainerStarted","Data":"3488eee92047d8ced889933e94fcd2a82f527c12868be793fc152aecb42a8b75"}
Oct 30 19:29:51.341546 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:51.341522    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr" event={"ID":"de8ddc3f-5d23-4cff-9f2e-9167700ff109","Type":"ContainerStarted","Data":"02463b2f1e9410da2914a91634756efcbe5c624a5fc7d98f8bf39b5c3c49d815"}
Oct 30 19:29:51.341627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:51.341586    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr" event={"ID":"de8ddc3f-5d23-4cff-9f2e-9167700ff109","Type":"ContainerStarted","Data":"a405226247d950e2d96eb74d8e0cbcce2cbf9f5523f9f93b0c6334fcd71e408d"}
Oct 30 19:29:51.341627 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:51.341615    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr" event={"ID":"de8ddc3f-5d23-4cff-9f2e-9167700ff109","Type":"ContainerStarted","Data":"16359519547f0e9d2ffea3dda6d72d176e331612cd50579629c250ecb10c1cf3"}
Oct 30 19:29:51.341684 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:51.341644    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr" event={"ID":"de8ddc3f-5d23-4cff-9f2e-9167700ff109","Type":"ContainerStarted","Data":"b9079985851de3214115f820faee4c13c04f86474e9f2fd776b38cc38b1efac7"}
Oct 30 19:29:51.367467 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:51.367392    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qmqjk" podStartSLOduration=6.734827625 podStartE2EDuration="17.367378333s" totalImagesPullingTime="10.632550708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:51.366915907 +0000 UTC m=+17.781554889" watchObservedRunningTime="2026-10-30 19:29:51.367378333 +0000 UTC m=+17.782017314"
Oct 30 19:29:51.647950 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:51.647862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs\") pod \"network-metrics-daemon-mwcmw\" (UID: \"7c979e36-8fff-4401-a991-1a661fd9c6e5\") " pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:51.648119 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:51.647957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:51.648119 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:51.648017    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:51.648119 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:51.648072    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:51.648119 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:51.648096    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs podName:7c979e36-8fff-4401-a991-1a661fd9c6e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:07.648080694 +0000 UTC m=+34.062719666 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs") pod "network-metrics-daemon-mwcmw" (UID: "7c979e36-8fff-4401-a991-1a661fd9c6e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:51.648119 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:51.648116    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret podName:8013e357-0fa2-4360-924b-18cbfe588b5c nodeName:}" failed. No retries permitted until 2026-10-30 19:30:07.648106236 +0000 UTC m=+34.062745210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret") pod "global-pull-secret-syncer-d2hv2" (UID: "8013e357-0fa2-4360-924b-18cbfe588b5c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:51.748663 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:51.748624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5z5s\" (UniqueName: \"kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s\") pod \"network-check-target-vwkv5\" (UID: \"101b32f8-ea50-4c3d-8114-15ebe26eb6d0\") " pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:51.748834 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:51.748744    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:29:51.748834 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:51.748759    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:29:51.748834 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:51.748767    2577 projected.go:196] Error preparing data for projected volume kube-api-access-d5z5s for pod openshift-network-diagnostics/network-check-target-vwkv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:51.748834 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:51.748812    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s podName:101b32f8-ea50-4c3d-8114-15ebe26eb6d0 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:07.748800537 +0000 UTC m=+34.163439498 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5z5s" (UniqueName: "kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s") pod "network-check-target-vwkv5" (UID: "101b32f8-ea50-4c3d-8114-15ebe26eb6d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:52.106988 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:52.106911    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:52.107136 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:52.107055    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:29:53.107541 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:53.107512    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:53.107967 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:53.107613    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:29:53.107967 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:53.107668    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:53.107967 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:53.107750    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:29:53.348245 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:53.348163    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr" event={"ID":"de8ddc3f-5d23-4cff-9f2e-9167700ff109","Type":"ContainerStarted","Data":"f5aa5e7064c58c6fa558d569fc3a61e4ab4b81391e4df102a16b8190d71f6df3"}
Oct 30 19:29:54.107856 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:54.107831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:54.108238 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:54.107906    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:29:55.107184 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:55.107155    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:55.107329 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:55.107271    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:29:55.107329 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:55.107318    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:55.107424 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:55.107406    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:29:55.359813 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:55.359557    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr" event={"ID":"de8ddc3f-5d23-4cff-9f2e-9167700ff109","Type":"ContainerStarted","Data":"9f88bedf3c1a8691ff45ea171c79d8338238bb7a78ab50572c755c4d921f9019"}
Oct 30 19:29:55.359813 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:55.359799    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:55.360364 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:55.359831    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:55.375042 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:55.375011    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:55.400676 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:55.400634    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr" podStartSLOduration=7.145804145 podStartE2EDuration="21.400621746s" totalImagesPullingTime="14.254817601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:55.400125819 +0000 UTC m=+21.814764800" watchObservedRunningTime="2026-10-30 19:29:55.400621746 +0000 UTC m=+21.815260788"
Oct 30 19:29:56.107500 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:56.107474    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:56.107675 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:56.107567    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:29:56.362478 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:56.362400    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:56.376449 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:56.376419    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:29:56.464509 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:56.464482    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kspfd"
Oct 30 19:29:56.464652 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:56.464610    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:29:56.465001 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:56.464984    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kspfd"
Oct 30 19:29:56.912022 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:56.911950    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d2hv2"]
Oct 30 19:29:56.912244 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:56.912090    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:56.912244 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:56.912198    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:29:56.914105 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:56.914080    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vwkv5"]
Oct 30 19:29:56.914210 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:56.914189    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:56.914322 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:56.914282    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:29:56.914580 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:56.914561    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mwcmw"]
Oct 30 19:29:56.914689 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:56.914675    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:56.914800 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:56.914780    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:29:57.477695 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:57.477669    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gmrbv_c2836676-0428-4f2f-a801-0a25df87aee2/dns-node-resolver/0.log"
Oct 30 19:29:58.394274 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:58.394246    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h5m52_53bd98ea-ea87-484a-b7b1-8c9070fc9b25/node-ca/0.log"
Oct 30 19:29:59.107823 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:59.107795    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:29:59.108245 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:59.107869    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:29:59.108245 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:29:59.107951    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:29:59.108245 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:59.107961    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:29:59.108245 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:59.108040    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:29:59.108245 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:29:59.108085    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:30:01.107019 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:01.106996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:30:01.107386 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:01.106996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:01.107386 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:01.107103    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:30:01.107386 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:01.107002    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:30:01.107386 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:01.107164    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:30:01.107386 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:01.107224    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:30:03.107045 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:03.106852    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:30:03.107373 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:03.106888    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:03.107373 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:03.107158    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:30:03.107373 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:03.106888    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:30:03.107373 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:03.107225    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:30:03.107373 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:03.107314    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:30:05.107517 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:05.107488    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:30:05.107904 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:05.107585    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:30:05.107904 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:05.107599    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:30:05.107904 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:05.107610    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:05.107904 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:05.107676    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:30:05.107904 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:05.107747    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:30:07.107274 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:07.107246    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:07.107626 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:07.107281    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:30:07.107626 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:07.107287    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:30:07.107626 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:07.107376    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:30:07.107626 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:07.107405    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:30:07.107626 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:07.107485    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:30:07.671847 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:07.671812    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:07.671847 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:07.671858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs\") pod \"network-metrics-daemon-mwcmw\" (UID: \"7c979e36-8fff-4401-a991-1a661fd9c6e5\") " pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:30:07.672050 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:07.671936    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:30:07.672050 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:07.671939    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:30:07.672050 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:07.671981    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs podName:7c979e36-8fff-4401-a991-1a661fd9c6e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:39.671967956 +0000 UTC m=+66.086606933 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs") pod "network-metrics-daemon-mwcmw" (UID: "7c979e36-8fff-4401-a991-1a661fd9c6e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:30:07.672050 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:07.671993    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret podName:8013e357-0fa2-4360-924b-18cbfe588b5c nodeName:}" failed. No retries permitted until 2026-10-30 19:30:39.671987839 +0000 UTC m=+66.086626800 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret") pod "global-pull-secret-syncer-d2hv2" (UID: "8013e357-0fa2-4360-924b-18cbfe588b5c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:30:07.772688 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:07.772660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5z5s\" (UniqueName: \"kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s\") pod \"network-check-target-vwkv5\" (UID: \"101b32f8-ea50-4c3d-8114-15ebe26eb6d0\") " pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:30:07.772835 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:07.772812    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:30:07.772835 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:07.772832    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:30:07.772946 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:07.772844    2577 projected.go:196] Error preparing data for projected volume kube-api-access-d5z5s for pod openshift-network-diagnostics/network-check-target-vwkv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:30:07.772946 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:07.772894    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s podName:101b32f8-ea50-4c3d-8114-15ebe26eb6d0 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:39.772878463 +0000 UTC m=+66.187517437 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5z5s" (UniqueName: "kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s") pod "network-check-target-vwkv5" (UID: "101b32f8-ea50-4c3d-8114-15ebe26eb6d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:30:09.107182 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:09.107152    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:30:09.107652 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:09.107152    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:09.107652 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:09.107289    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:30:09.107652 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:09.107152    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:30:09.107652 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:09.107317    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:30:09.107652 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:09.107384    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:30:11.107778 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:11.107747    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:30:11.107778 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:11.107780    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:11.107778 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:11.107779    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:30:11.108230 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:11.107853    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:30:11.108230 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:11.107958    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:30:11.108230 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:11.108020    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:30:13.107373 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:13.107343    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:13.107756 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:13.107439    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:30:13.107756 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:13.107459    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:30:13.107756 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:13.107536    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:30:13.107756 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:13.107459    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:30:13.107756 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:13.107606    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:30:15.107512 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:15.107488    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:30:15.107791 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:15.107553    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:30:15.107791 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:15.107579    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:15.107791 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:15.107645    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:30:15.107791 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:15.107691    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:30:15.107911 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:15.107784    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:30:17.107571 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.107544    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:17.107571 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.107567    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:30:17.108050 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.107544    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:30:17.108050 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:17.107659    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2hv2" podUID="8013e357-0fa2-4360-924b-18cbfe588b5c"
Oct 30 19:30:17.108050 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:17.107725    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwcmw" podUID="7c979e36-8fff-4401-a991-1a661fd9c6e5"
Oct 30 19:30:17.108050 ip-10-0-141-19 kubenswrapper[2577]: E1030 19:30:17.107780    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vwkv5" podUID="101b32f8-ea50-4c3d-8114-15ebe26eb6d0"
Oct 30 19:30:17.394498 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.394460    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeReady"
Oct 30 19:30:17.394655 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.394564    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 19:30:17.447928 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.447897    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2c2l4"]
Oct 30 19:30:17.450045 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.450017    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2c2l4"
Oct 30 19:30:17.454574 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.454553    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rkc6x"]
Oct 30 19:30:17.456322 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.456298    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:17.467003 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.466980    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:30:17.467098 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.467050    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ttn9c\""
Oct 30 19:30:17.467356 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.467326    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:30:17.467454 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.467402    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:30:17.471360 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.471342    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:30:17.472131 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.472110    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2c2l4"]
Oct 30 19:30:17.479573 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.479553    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dkkwh"]
Oct 30 19:30:17.481321 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.481275    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rkc6x"]
Oct 30 19:30:17.481408 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.481399    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:17.496615 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.496598    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dkkwh"]
Oct 30 19:30:17.552023 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.551994    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxxqw\" (UniqueName: \"kubernetes.io/projected/0de2915a-789c-4acc-84a6-c61987498b62-kube-api-access-nxxqw\") pod \"ingress-canary-2c2l4\" (UID: \"0de2915a-789c-4acc-84a6-c61987498b62\") " pod="openshift-ingress-canary/ingress-canary-2c2l4"
Oct 30 19:30:17.552166 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.552053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0de2915a-789c-4acc-84a6-c61987498b62-cert\") pod \"ingress-canary-2c2l4\" (UID: \"0de2915a-789c-4acc-84a6-c61987498b62\") " pod="openshift-ingress-canary/ingress-canary-2c2l4"
Oct 30 19:30:17.566376 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.566352    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:30:17.566376 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.566372    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:30:17.566547 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.566506    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:30:17.566547 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.566525    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:30:17.566547 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.566538    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:30:17.566653 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.566541    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zsvdk\""
Oct 30 19:30:17.566653 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.566547    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-svfmj\""
Oct 30 19:30:17.653215 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.653161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-metrics-tls\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:17.653215 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.653202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-config-volume\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:17.653345 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.653224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5sd9\" (UniqueName: \"kubernetes.io/projected/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-kube-api-access-q5sd9\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:17.653345 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.653283    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45ccae2d-1b91-4bbe-9598-94ad740116ec-crio-socket\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:17.653345 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.653337    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-tmp-dir\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:17.653443 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.653400    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxxqw\" (UniqueName: \"kubernetes.io/projected/0de2915a-789c-4acc-84a6-c61987498b62-kube-api-access-nxxqw\") pod \"ingress-canary-2c2l4\" (UID: \"0de2915a-789c-4acc-84a6-c61987498b62\") " pod="openshift-ingress-canary/ingress-canary-2c2l4"
Oct 30 19:30:17.653443 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.653424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45ccae2d-1b91-4bbe-9598-94ad740116ec-data-volume\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:17.653512 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.653490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5sb5\" (UniqueName: \"kubernetes.io/projected/45ccae2d-1b91-4bbe-9598-94ad740116ec-kube-api-access-d5sb5\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:17.653545 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.653518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0de2915a-789c-4acc-84a6-c61987498b62-cert\") pod \"ingress-canary-2c2l4\" (UID: \"0de2915a-789c-4acc-84a6-c61987498b62\") " pod="openshift-ingress-canary/ingress-canary-2c2l4"
Oct 30 19:30:17.653581 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.653551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45ccae2d-1b91-4bbe-9598-94ad740116ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:17.653613 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.653574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45ccae2d-1b91-4bbe-9598-94ad740116ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:17.657238 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.657215    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0de2915a-789c-4acc-84a6-c61987498b62-cert\") pod \"ingress-canary-2c2l4\" (UID: \"0de2915a-789c-4acc-84a6-c61987498b62\") " pod="openshift-ingress-canary/ingress-canary-2c2l4"
Oct 30 19:30:17.666314 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.666293    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxxqw\" (UniqueName: \"kubernetes.io/projected/0de2915a-789c-4acc-84a6-c61987498b62-kube-api-access-nxxqw\") pod \"ingress-canary-2c2l4\" (UID: \"0de2915a-789c-4acc-84a6-c61987498b62\") " pod="openshift-ingress-canary/ingress-canary-2c2l4"
Oct 30 19:30:17.754723 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.754701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45ccae2d-1b91-4bbe-9598-94ad740116ec-crio-socket\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:17.754805 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.754729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-tmp-dir\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:17.754805 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.754754    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45ccae2d-1b91-4bbe-9598-94ad740116ec-data-volume\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:17.754871 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.754849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5sb5\" (UniqueName: \"kubernetes.io/projected/45ccae2d-1b91-4bbe-9598-94ad740116ec-kube-api-access-d5sb5\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:17.754902 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.754887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45ccae2d-1b91-4bbe-9598-94ad740116ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:17.754942 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.754902    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45ccae2d-1b91-4bbe-9598-94ad740116ec-crio-socket\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:17.754942 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.754913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45ccae2d-1b91-4bbe-9598-94ad740116ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:17.755062 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.754975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-metrics-tls\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:17.755062 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.755007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-config-volume\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:17.755062 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.755047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5sd9\" (UniqueName: \"kubernetes.io/projected/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-kube-api-access-q5sd9\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:17.755203 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.755078    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45ccae2d-1b91-4bbe-9598-94ad740116ec-data-volume\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:17.755203 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.755079    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-tmp-dir\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:17.755497 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.755474    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45ccae2d-1b91-4bbe-9598-94ad740116ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:17.755576 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.755518    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-config-volume\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:17.756866 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.756844    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-metrics-tls\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:17.757088 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.757070    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45ccae2d-1b91-4bbe-9598-94ad740116ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:17.761037 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.761009    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2c2l4"
Oct 30 19:30:17.768210 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.768190    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5sb5\" (UniqueName: \"kubernetes.io/projected/45ccae2d-1b91-4bbe-9598-94ad740116ec-kube-api-access-d5sb5\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:17.768301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.768263    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5sd9\" (UniqueName: \"kubernetes.io/projected/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-kube-api-access-q5sd9\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:17.770115 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.770099    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:17.790727 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:17.790706    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:18.060931 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.060856    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2c2l4"]
Oct 30 19:30:18.064459 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.064436    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rkc6x"]
Oct 30 19:30:18.065750 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:30:18.065724    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de2915a_789c_4acc_84a6_c61987498b62.slice/crio-47ee8ad7afa297228f1b3adfeb887e1799deacaaad1a2cabc351691e1f0a39c5 WatchSource:0}: Error finding container 47ee8ad7afa297228f1b3adfeb887e1799deacaaad1a2cabc351691e1f0a39c5: Status 404 returned error can't find the container with id 47ee8ad7afa297228f1b3adfeb887e1799deacaaad1a2cabc351691e1f0a39c5
Oct 30 19:30:18.066940 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:30:18.066923    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ccae2d_1b91_4bbe_9598_94ad740116ec.slice/crio-771d5ecbcb285260e210781715caeda168dd97a9820a0dfb0cb9a687f8a1c7ec WatchSource:0}: Error finding container 771d5ecbcb285260e210781715caeda168dd97a9820a0dfb0cb9a687f8a1c7ec: Status 404 returned error can't find the container with id 771d5ecbcb285260e210781715caeda168dd97a9820a0dfb0cb9a687f8a1c7ec
Oct 30 19:30:18.076926 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.076902    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dkkwh"]
Oct 30 19:30:18.079766 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:30:18.079745    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e933e1_4e6d_4a26_bb7d_68f366e02ac5.slice/crio-f4d92d32030cb453da5f69397058a394d5892d1066c08079fb94644e86839df7 WatchSource:0}: Error finding container f4d92d32030cb453da5f69397058a394d5892d1066c08079fb94644e86839df7: Status 404 returned error can't find the container with id f4d92d32030cb453da5f69397058a394d5892d1066c08079fb94644e86839df7
Oct 30 19:30:18.129161 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.129137    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-854b9d69db-nvllp"]
Oct 30 19:30:18.131446 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.131429    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.136839 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.136811    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 19:30:18.137018 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.137000    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 19:30:18.137129 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.137054    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 19:30:18.138105 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.138088    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 19:30:18.138193 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.138104    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6jgjw\""
Oct 30 19:30:18.138193 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.138137    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 19:30:18.138193 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.138154    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 19:30:18.138297 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.138209    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 19:30:18.145815 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.145722    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-854b9d69db-nvllp"]
Oct 30 19:30:18.158150 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.158131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-oauth-serving-cert\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.158264 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.158167    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-serving-cert\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.158264 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.158185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-oauth-config\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.158370 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.158296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-config\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.158370 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.158330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e9972f4-8918-41a0-8ae4-b2014d3897ca-tmp\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.158370 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.158360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4btm\" (UniqueName: \"kubernetes.io/projected/3e9972f4-8918-41a0-8ae4-b2014d3897ca-kube-api-access-b4btm\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.158509 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.158476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-service-ca\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.259299 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.259273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-config\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.259438 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.259303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e9972f4-8918-41a0-8ae4-b2014d3897ca-tmp\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.259438 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.259320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4btm\" (UniqueName: \"kubernetes.io/projected/3e9972f4-8918-41a0-8ae4-b2014d3897ca-kube-api-access-b4btm\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.259438 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.259339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-service-ca\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.259438 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.259371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-oauth-serving-cert\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.259438 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.259398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-serving-cert\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.259438 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.259419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-oauth-config\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.259759 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.259740    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e9972f4-8918-41a0-8ae4-b2014d3897ca-tmp\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.260125 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.260098    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-config\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.260125 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.260113    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-service-ca\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.260267 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.260158    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-oauth-serving-cert\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.261758 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.261737    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-oauth-config\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.261837 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.261805    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-serving-cert\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.275470 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.275448    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4btm\" (UniqueName: \"kubernetes.io/projected/3e9972f4-8918-41a0-8ae4-b2014d3897ca-kube-api-access-b4btm\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.406536 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.406442    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dkkwh" event={"ID":"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5","Type":"ContainerStarted","Data":"f4d92d32030cb453da5f69397058a394d5892d1066c08079fb94644e86839df7"}
Oct 30 19:30:18.407864 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.407846    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rkc6x" event={"ID":"45ccae2d-1b91-4bbe-9598-94ad740116ec","Type":"ContainerStarted","Data":"817df5832bc2b09eeefaf1019fcf1fcdbd210f7943b3c3a1327dc9e2ea4d2623"}
Oct 30 19:30:18.407933 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.407885    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rkc6x" event={"ID":"45ccae2d-1b91-4bbe-9598-94ad740116ec","Type":"ContainerStarted","Data":"771d5ecbcb285260e210781715caeda168dd97a9820a0dfb0cb9a687f8a1c7ec"}
Oct 30 19:30:18.408817 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.408794    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2c2l4" event={"ID":"0de2915a-789c-4acc-84a6-c61987498b62","Type":"ContainerStarted","Data":"47ee8ad7afa297228f1b3adfeb887e1799deacaaad1a2cabc351691e1f0a39c5"}
Oct 30 19:30:18.441990 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.441968    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:18.721002 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:18.720888    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-854b9d69db-nvllp"]
Oct 30 19:30:18.725675 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:30:18.725645    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9972f4_8918_41a0_8ae4_b2014d3897ca.slice/crio-e59da0f23b582ab57c0b73f7043466ba70b4e7e2b61094e81723590699ea17f1 WatchSource:0}: Error finding container e59da0f23b582ab57c0b73f7043466ba70b4e7e2b61094e81723590699ea17f1: Status 404 returned error can't find the container with id e59da0f23b582ab57c0b73f7043466ba70b4e7e2b61094e81723590699ea17f1
Oct 30 19:30:19.107893 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.107829    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:19.107893 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.107845    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:30:19.107893 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.107856    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:30:19.113214 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.113192    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:30:19.114427 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.114407    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2prmp\""
Oct 30 19:30:19.114574 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.114464    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:30:19.114574 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.114551    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-q9kwd\""
Oct 30 19:30:19.114754 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.114659    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:30:19.114754 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.114694    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:30:19.114856 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.114781    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8hwq2\""
Oct 30 19:30:19.413956 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.413876    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854b9d69db-nvllp" event={"ID":"3e9972f4-8918-41a0-8ae4-b2014d3897ca","Type":"ContainerStarted","Data":"e59da0f23b582ab57c0b73f7043466ba70b4e7e2b61094e81723590699ea17f1"}
Oct 30 19:30:19.416106 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.416079    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rkc6x" event={"ID":"45ccae2d-1b91-4bbe-9598-94ad740116ec","Type":"ContainerStarted","Data":"7c6e55ed37a3d413e101b8d9fda68f479314e1a82d82d755b70ee842fd2127a7"}
Oct 30 19:30:19.690923 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.690851    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"]
Oct 30 19:30:19.692738 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.692705    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:19.706455 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.706431    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 19:30:19.706573 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.706550    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:30:19.707962 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.707943    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 19:30:19.708087 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.708044    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nr7cs\""
Oct 30 19:30:19.708160 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.708106    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:30:19.708160 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.708134    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 19:30:19.709164 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.709128    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"]
Oct 30 19:30:19.773700 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.773669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008928f8-b2ce-4f12-9681-21136d802ff6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:19.773852 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.773733    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/008928f8-b2ce-4f12-9681-21136d802ff6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:19.773852 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.773766    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8bxp\" (UniqueName: \"kubernetes.io/projected/008928f8-b2ce-4f12-9681-21136d802ff6-kube-api-access-j8bxp\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:19.773852 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.773791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/008928f8-b2ce-4f12-9681-21136d802ff6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:19.874225 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.874197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008928f8-b2ce-4f12-9681-21136d802ff6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:19.874405 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.874256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/008928f8-b2ce-4f12-9681-21136d802ff6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:19.874405 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.874278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8bxp\" (UniqueName: \"kubernetes.io/projected/008928f8-b2ce-4f12-9681-21136d802ff6-kube-api-access-j8bxp\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:19.874405 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.874297    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/008928f8-b2ce-4f12-9681-21136d802ff6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:19.874954 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.874931    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008928f8-b2ce-4f12-9681-21136d802ff6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:19.880273 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.880221    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/008928f8-b2ce-4f12-9681-21136d802ff6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:19.881826 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.881786    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/008928f8-b2ce-4f12-9681-21136d802ff6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:19.888867 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:19.888823    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8bxp\" (UniqueName: \"kubernetes.io/projected/008928f8-b2ce-4f12-9681-21136d802ff6-kube-api-access-j8bxp\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:20.008109 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:20.007835    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:20.311782 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:20.311753    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"]
Oct 30 19:30:20.324187 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:30:20.324148    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008928f8_b2ce_4f12_9681_21136d802ff6.slice/crio-4cda2bf1ae154e1b02cf339cbcb9249ff3eb3aa791a8213ba323612fa10282cc WatchSource:0}: Error finding container 4cda2bf1ae154e1b02cf339cbcb9249ff3eb3aa791a8213ba323612fa10282cc: Status 404 returned error can't find the container with id 4cda2bf1ae154e1b02cf339cbcb9249ff3eb3aa791a8213ba323612fa10282cc
Oct 30 19:30:20.421666 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:20.421637    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l" event={"ID":"008928f8-b2ce-4f12-9681-21136d802ff6","Type":"ContainerStarted","Data":"4cda2bf1ae154e1b02cf339cbcb9249ff3eb3aa791a8213ba323612fa10282cc"}
Oct 30 19:30:20.424557 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:20.424536    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dkkwh" event={"ID":"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5","Type":"ContainerStarted","Data":"c0868c5392ba6b93c56ca869b193e9c60eb2ccdceb67fc4862a4ff2a8c4bae3b"}
Oct 30 19:30:21.430078 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:21.430023    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2c2l4" event={"ID":"0de2915a-789c-4acc-84a6-c61987498b62","Type":"ContainerStarted","Data":"8fc0f3365911b133da6b44681bc5e9be1ce8d3ad672bdacf01c8a8d50a05ac60"}
Oct 30 19:30:21.432550 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:21.432518    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dkkwh" event={"ID":"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5","Type":"ContainerStarted","Data":"f6268fdeecd6dc2767f8c99c1283dd17438a6b110562d917dff6708b9ccd3f8f"}
Oct 30 19:30:21.432690 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:21.432613    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:21.435264 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:21.435206    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rkc6x" event={"ID":"45ccae2d-1b91-4bbe-9598-94ad740116ec","Type":"ContainerStarted","Data":"c0502ec442a12c0d25ff44eb2938ab4deca142934a09ff5a8fbf1a3dd906fa06"}
Oct 30 19:30:21.483357 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:21.478363    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2c2l4" podStartSLOduration=1.228128598 podStartE2EDuration="4.478344919s" totalImagesPullingTime="3.250216321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:21.45111509 +0000 UTC m=+47.865754090" watchObservedRunningTime="2026-10-30 19:30:21.478344919 +0000 UTC m=+47.892983902"
Oct 30 19:30:21.502132 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:21.502067    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rkc6x" podStartSLOduration=1.900573394 podStartE2EDuration="4.502049551s" totalImagesPullingTime="2.601476157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:21.479944202 +0000 UTC m=+47.894583217" watchObservedRunningTime="2026-10-30 19:30:21.502049551 +0000 UTC m=+47.916688535"
Oct 30 19:30:21.503014 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:21.502982    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-dkkwh" podStartSLOduration=2.395044228 podStartE2EDuration="4.502970573s" totalImagesPullingTime="2.107926345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:21.50149259 +0000 UTC m=+47.916131573" watchObservedRunningTime="2026-10-30 19:30:21.502970573 +0000 UTC m=+47.917609554"
Oct 30 19:30:22.440104 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:22.440068    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l" event={"ID":"008928f8-b2ce-4f12-9681-21136d802ff6","Type":"ContainerStarted","Data":"29634bd30d74b5e241937c9d5f37ebb0509f7cfa7872816ef5e1e02a46ad612c"}
Oct 30 19:30:22.440547 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:22.440127    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l" event={"ID":"008928f8-b2ce-4f12-9681-21136d802ff6","Type":"ContainerStarted","Data":"f8c80cd356a0d0532a830eca31d5edeb9cebeacb5534c269f0b7bbc93a06a4d5"}
Oct 30 19:30:22.441605 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:22.441575    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854b9d69db-nvllp" event={"ID":"3e9972f4-8918-41a0-8ae4-b2014d3897ca","Type":"ContainerStarted","Data":"48e72b4766f1ddcf4e665db835b405128feeabc1771724d7fca3cc955561de47"}
Oct 30 19:30:22.481092 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:22.481023    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l" podStartSLOduration=2.1145183850000002 podStartE2EDuration="3.481006067s" totalImagesPullingTime="1.366487682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:22.460203825 +0000 UTC m=+48.874842830" watchObservedRunningTime="2026-10-30 19:30:22.481006067 +0000 UTC m=+48.895645051"
Oct 30 19:30:22.495795 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:22.495739    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-854b9d69db-nvllp" podStartSLOduration=1.033417826 podStartE2EDuration="4.495722405s" totalImagesPullingTime="3.462304579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:22.495408604 +0000 UTC m=+48.910047610" watchObservedRunningTime="2026-10-30 19:30:22.495722405 +0000 UTC m=+48.910361387"
Oct 30 19:30:24.157373 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.157342    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rb876"]
Oct 30 19:30:24.161905 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.161881    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.177843 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.177815    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 19:30:24.178022 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.178006    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ch4qh\""
Oct 30 19:30:24.178177 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.178157    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 19:30:24.211837 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.211810    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-sys\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.211957 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.211885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z7pj\" (UniqueName: \"kubernetes.io/projected/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-kube-api-access-7z7pj\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.212000 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.211968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-tls\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.212053 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.212013    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-textfile\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.212094 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.212075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-accelerators-collector-config\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.212127 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.212106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-wtmp\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.212170 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.212128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-metrics-client-ca\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.212266 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.212247    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-root\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.212311 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.212275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.254674 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.254652    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 19:30:24.313620 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.313593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z7pj\" (UniqueName: \"kubernetes.io/projected/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-kube-api-access-7z7pj\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.313727 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.313623    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-tls\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.313727 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.313660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-textfile\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.313812 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.313792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-accelerators-collector-config\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.313846 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.313836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-wtmp\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.313884 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.313864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-metrics-client-ca\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.313954 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.313939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-root\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.314002 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.313969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.314079 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.314008    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-textfile\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.314079 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.314016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-sys\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.314172 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.314072    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-root\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.314172 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.314085    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-sys\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.314172 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.314017    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-wtmp\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.314441 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.314422    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-metrics-client-ca\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.314441 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.314430    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-accelerators-collector-config\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.316053 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.316024    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.316259 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.316237    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-tls\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.331307 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.331284    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z7pj\" (UniqueName: \"kubernetes.io/projected/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-kube-api-access-7z7pj\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.478287 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:24.478215    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:24.488860 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:30:24.488836    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbc20d7_e495_4d90_8c4c_a2fcc76a0baf.slice/crio-823e2f643e9b2f48bd55729c3f8353509829c76f10b6e67696ff3b2964280b0f WatchSource:0}: Error finding container 823e2f643e9b2f48bd55729c3f8353509829c76f10b6e67696ff3b2964280b0f: Status 404 returned error can't find the container with id 823e2f643e9b2f48bd55729c3f8353509829c76f10b6e67696ff3b2964280b0f
Oct 30 19:30:25.451751 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:25.451720    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rb876" event={"ID":"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf","Type":"ContainerStarted","Data":"823e2f643e9b2f48bd55729c3f8353509829c76f10b6e67696ff3b2964280b0f"}
Oct 30 19:30:26.282359 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.282324    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-67c75b6564-g8dq2"]
Oct 30 19:30:26.285707 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.285692    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.299709 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.299681    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67c75b6564-g8dq2"]
Oct 30 19:30:26.305211 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.305161    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 19:30:26.305354 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.305336    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 19:30:26.305422 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.305339    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 19:30:26.305619 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.305606    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 19:30:26.305619 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.305617    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bl6kg3usqeas\""
Oct 30 19:30:26.305720 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.305712    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tr8z5\""
Oct 30 19:30:26.305753 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.305710    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 19:30:26.329315 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.329290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.329449 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.329336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.329449 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.329355    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0d19f3a-000a-4911-96dd-cbf24e784627-tmp\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.329449 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.329426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.329564 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.329454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.329564 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.329536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d19f3a-000a-4911-96dd-cbf24e784627-metrics-client-ca\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.329626 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.329574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-grpc-tls\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.329626 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.329611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-tls\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.329692 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.329679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5rj9\" (UniqueName: \"kubernetes.io/projected/e0d19f3a-000a-4911-96dd-cbf24e784627-kube-api-access-w5rj9\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.430954 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.430915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.431133 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.430962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.431133 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.430979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0d19f3a-000a-4911-96dd-cbf24e784627-tmp\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.431133 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.431012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.431133 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.431046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.431133 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.431091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d19f3a-000a-4911-96dd-cbf24e784627-metrics-client-ca\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.431133 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.431115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-grpc-tls\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.431437 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.431148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-tls\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.431437 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.431184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5rj9\" (UniqueName: \"kubernetes.io/projected/e0d19f3a-000a-4911-96dd-cbf24e784627-kube-api-access-w5rj9\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.431540 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.431519    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0d19f3a-000a-4911-96dd-cbf24e784627-tmp\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.432267 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.432236    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d19f3a-000a-4911-96dd-cbf24e784627-metrics-client-ca\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.433883 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.433856    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-grpc-tls\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.433981 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.433868    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-tls\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.434023 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.433988    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.434184 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.434162    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.434218 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.434162    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.434290 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.434263    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.443880 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.443859    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5rj9\" (UniqueName: \"kubernetes.io/projected/e0d19f3a-000a-4911-96dd-cbf24e784627-kube-api-access-w5rj9\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.456145 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.456119    2577 generic.go:356] "Generic (PLEG): container finished" podID="0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf" containerID="026a7c68a8b24384c1d684c5e8bf908d0746343d8f6ad743e5440db8dc719b2c" exitCode=0
Oct 30 19:30:26.456431 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.456153    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rb876" event={"ID":"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf","Type":"ContainerDied","Data":"026a7c68a8b24384c1d684c5e8bf908d0746343d8f6ad743e5440db8dc719b2c"}
Oct 30 19:30:26.607658 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.607625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:26.888841 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:26.888770    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67c75b6564-g8dq2"]
Oct 30 19:30:26.891441 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:30:26.891416    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d19f3a_000a_4911_96dd_cbf24e784627.slice/crio-3e80dc474dd09ed7516ce7158907a43843349e37465b9e0ab5ff913cfce5c9ce WatchSource:0}: Error finding container 3e80dc474dd09ed7516ce7158907a43843349e37465b9e0ab5ff913cfce5c9ce: Status 404 returned error can't find the container with id 3e80dc474dd09ed7516ce7158907a43843349e37465b9e0ab5ff913cfce5c9ce
Oct 30 19:30:27.460160 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:27.460127    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2" event={"ID":"e0d19f3a-000a-4911-96dd-cbf24e784627","Type":"ContainerStarted","Data":"3e80dc474dd09ed7516ce7158907a43843349e37465b9e0ab5ff913cfce5c9ce"}
Oct 30 19:30:27.462135 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:27.462113    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rb876" event={"ID":"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf","Type":"ContainerStarted","Data":"8fa596071194f3283fae87d232dd4242212f719a998320cb726cb26d964be79b"}
Oct 30 19:30:27.462291 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:27.462151    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rb876" event={"ID":"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf","Type":"ContainerStarted","Data":"7c089a95a84696d8a837dec5700b598535eee43b7d18bee8fae812eedaedc7fd"}
Oct 30 19:30:27.490902 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:27.490860    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rb876" podStartSLOduration=2.5118533259999998 podStartE2EDuration="3.490847584s" totalImagesPullingTime="912.335519ms" totalInitContainerRuntime="66.658739ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:27.488596232 +0000 UTC m=+53.903235217" watchObservedRunningTime="2026-10-30 19:30:27.490847584 +0000 UTC m=+53.905486567"
Oct 30 19:30:28.376703 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:28.376670    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:28.443007 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:28.442976    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:28.443217 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:28.443064    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:28.448279 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:28.448256    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:28.469841 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:28.469820    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:28.938942 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:28.938922    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7"]
Oct 30 19:30:28.941885 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:28.941869    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7"
Oct 30 19:30:28.953094 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:28.953071    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 19:30:28.953185 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:28.953145    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-f48l9\""
Oct 30 19:30:28.953873 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:28.953854    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7"]
Oct 30 19:30:29.053165 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:29.053140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a341b7ef-207b-4443-b01e-9fcbc2ac0075-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ldlt7\" (UID: \"a341b7ef-207b-4443-b01e-9fcbc2ac0075\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7"
Oct 30 19:30:29.053301 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:29.053190    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d895r\" (UniqueName: \"kubernetes.io/projected/a341b7ef-207b-4443-b01e-9fcbc2ac0075-kube-api-access-d895r\") pod \"monitoring-plugin-77f4f76594-ldlt7\" (UID: \"a341b7ef-207b-4443-b01e-9fcbc2ac0075\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7"
Oct 30 19:30:29.154374 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:29.154295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a341b7ef-207b-4443-b01e-9fcbc2ac0075-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ldlt7\" (UID: \"a341b7ef-207b-4443-b01e-9fcbc2ac0075\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7"
Oct 30 19:30:29.154374 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:29.154366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d895r\" (UniqueName: \"kubernetes.io/projected/a341b7ef-207b-4443-b01e-9fcbc2ac0075-kube-api-access-d895r\") pod \"monitoring-plugin-77f4f76594-ldlt7\" (UID: \"a341b7ef-207b-4443-b01e-9fcbc2ac0075\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7"
Oct 30 19:30:29.156706 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:29.156678    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a341b7ef-207b-4443-b01e-9fcbc2ac0075-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ldlt7\" (UID: \"a341b7ef-207b-4443-b01e-9fcbc2ac0075\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7"
Oct 30 19:30:29.167377 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:29.167353    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d895r\" (UniqueName: \"kubernetes.io/projected/a341b7ef-207b-4443-b01e-9fcbc2ac0075-kube-api-access-d895r\") pod \"monitoring-plugin-77f4f76594-ldlt7\" (UID: \"a341b7ef-207b-4443-b01e-9fcbc2ac0075\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7"
Oct 30 19:30:29.252290 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:29.252264    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7"
Oct 30 19:30:29.470844 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:29.470774    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2" event={"ID":"e0d19f3a-000a-4911-96dd-cbf24e784627","Type":"ContainerStarted","Data":"c4c01abd496c70896e1a4f9f0314bb9ab460721381e316bae2fb78bd60a7e2aa"}
Oct 30 19:30:29.470844 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:29.470840    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2" event={"ID":"e0d19f3a-000a-4911-96dd-cbf24e784627","Type":"ContainerStarted","Data":"9ec926bbff76ab428a2b419f8e7bbfe85522acd465e4a427907d201a46028721"}
Oct 30 19:30:29.471313 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:29.470869    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2" event={"ID":"e0d19f3a-000a-4911-96dd-cbf24e784627","Type":"ContainerStarted","Data":"658feca98e8af330df821f4357446062c331318336ca07ea4dcbe677ab5e6bb2"}
Oct 30 19:30:29.524268 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:29.524242    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7"]
Oct 30 19:30:29.527962 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:30:29.527937    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda341b7ef_207b_4443_b01e_9fcbc2ac0075.slice/crio-1f3a89bbe93fdc66236bd2765050de4d26d191fef04c0713b85c04d3b08a37fc WatchSource:0}: Error finding container 1f3a89bbe93fdc66236bd2765050de4d26d191fef04c0713b85c04d3b08a37fc: Status 404 returned error can't find the container with id 1f3a89bbe93fdc66236bd2765050de4d26d191fef04c0713b85c04d3b08a37fc
Oct 30 19:30:30.476354 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:30.476318    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7" event={"ID":"a341b7ef-207b-4443-b01e-9fcbc2ac0075","Type":"ContainerStarted","Data":"1f3a89bbe93fdc66236bd2765050de4d26d191fef04c0713b85c04d3b08a37fc"}
Oct 30 19:30:30.479619 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:30.479590    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2" event={"ID":"e0d19f3a-000a-4911-96dd-cbf24e784627","Type":"ContainerStarted","Data":"aa80ab0f31f9469dea85580b5e49ff5e5997e564df85a6d97befdf6ea6cb6e5c"}
Oct 30 19:30:30.479750 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:30.479699    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2" event={"ID":"e0d19f3a-000a-4911-96dd-cbf24e784627","Type":"ContainerStarted","Data":"605b8c69732f3002aa8d08fba68e44ce9f6e84da9fb19c2bc07318a64db2bf2c"}
Oct 30 19:30:30.479816 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:30.479750    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2" event={"ID":"e0d19f3a-000a-4911-96dd-cbf24e784627","Type":"ContainerStarted","Data":"1cd1b84eba2ca02a2ca9e50fdd929ed906a6ff41678fe12e35d5f830b33d20e4"}
Oct 30 19:30:30.514008 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:30.513954    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2" podStartSLOduration=1.802136671 podStartE2EDuration="4.51393638s" totalImagesPullingTime="2.711799709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:30.510008767 +0000 UTC m=+56.924647767" watchObservedRunningTime="2026-10-30 19:30:30.51393638 +0000 UTC m=+56.928575363"
Oct 30 19:30:31.444242 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:31.444212    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:31.485727 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:31.485693    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7" event={"ID":"a341b7ef-207b-4443-b01e-9fcbc2ac0075","Type":"ContainerStarted","Data":"7f006466706393ebf5a21286c45f3688362d848a97006aa8fec796288ab46fbf"}
Oct 30 19:30:31.486372 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:31.486352    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:31.486487 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:31.486415    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7"
Oct 30 19:30:31.491172 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:31.491148    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7"
Oct 30 19:30:31.505624 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:31.505581    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7" podStartSLOduration=2.077744824 podStartE2EDuration="3.505569123s" totalImagesPullingTime="1.427824299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:31.50428415 +0000 UTC m=+57.918923133" watchObservedRunningTime="2026-10-30 19:30:31.505569123 +0000 UTC m=+57.920208105"
Oct 30 19:30:32.979097 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:32.979061    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-698967b46-ghthb"]
Oct 30 19:30:32.982144 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:32.982121    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:32.995413 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:32.995387    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 19:30:32.997383 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:32.997363    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-698967b46-ghthb"]
Oct 30 19:30:33.090681 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.090647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-trusted-ca-bundle\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.090681 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.090686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-serving-cert\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.090889 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.090701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-oauth-config\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.090889 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.090841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k7xh\" (UniqueName: \"kubernetes.io/projected/c112dbe2-b2e4-4762-8123-b9f161bda84a-kube-api-access-6k7xh\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.090964 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.090899    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-oauth-serving-cert\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.090964 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.090928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-service-ca\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.091073 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.091011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c112dbe2-b2e4-4762-8123-b9f161bda84a-tmp\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.091111 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.091078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-config\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.191779 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.191738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k7xh\" (UniqueName: \"kubernetes.io/projected/c112dbe2-b2e4-4762-8123-b9f161bda84a-kube-api-access-6k7xh\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.191966 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.191805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-oauth-serving-cert\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.191966 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.191836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-service-ca\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.191966 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.191856    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c112dbe2-b2e4-4762-8123-b9f161bda84a-tmp\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.191966 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.191884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-config\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.191966 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.191927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-trusted-ca-bundle\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.191966 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.191951    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-serving-cert\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.192267 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.191973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-oauth-config\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.192384 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.192360    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c112dbe2-b2e4-4762-8123-b9f161bda84a-tmp\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.192715 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.192691    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-service-ca\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.192831 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.192772    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-oauth-serving-cert\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.192831 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.192811    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-config\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.192993 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.192968    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-trusted-ca-bundle\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.194511 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.194486    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-oauth-config\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.194583 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.194548    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-serving-cert\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.204924 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.204901    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k7xh\" (UniqueName: \"kubernetes.io/projected/c112dbe2-b2e4-4762-8123-b9f161bda84a-kube-api-access-6k7xh\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.293236 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.293198    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:33.571472 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:33.571401    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-698967b46-ghthb"]
Oct 30 19:30:33.575063 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:30:33.575012    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc112dbe2_b2e4_4762_8123_b9f161bda84a.slice/crio-c326058e8f2e9a4ae0387a68eab1d87ddbdf11405269f074395b5f275417f1c6 WatchSource:0}: Error finding container c326058e8f2e9a4ae0387a68eab1d87ddbdf11405269f074395b5f275417f1c6: Status 404 returned error can't find the container with id c326058e8f2e9a4ae0387a68eab1d87ddbdf11405269f074395b5f275417f1c6
Oct 30 19:30:34.497572 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:34.497536    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-698967b46-ghthb" event={"ID":"c112dbe2-b2e4-4762-8123-b9f161bda84a","Type":"ContainerStarted","Data":"75df4c23c8e2de0673867ccc295fd57cded72f617ce7744def6f652591a3a45d"}
Oct 30 19:30:34.497963 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:34.497585    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-698967b46-ghthb" event={"ID":"c112dbe2-b2e4-4762-8123-b9f161bda84a","Type":"ContainerStarted","Data":"c326058e8f2e9a4ae0387a68eab1d87ddbdf11405269f074395b5f275417f1c6"}
Oct 30 19:30:34.525237 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:34.525185    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-698967b46-ghthb" podStartSLOduration=2.525166386 podStartE2EDuration="2.525166386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:34.523466078 +0000 UTC m=+60.938105060" watchObservedRunningTime="2026-10-30 19:30:34.525166386 +0000 UTC m=+60.939805369"
Oct 30 19:30:37.495310 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:37.495284    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:39.746672 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:39.746639    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs\") pod \"network-metrics-daemon-mwcmw\" (UID: \"7c979e36-8fff-4401-a991-1a661fd9c6e5\") " pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:30:39.747229 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:39.746698    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:39.750316 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:39.750296    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:30:39.751921 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:39.751908    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:30:39.759268 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:39.759237    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs\") pod \"network-metrics-daemon-mwcmw\" (UID: \"7c979e36-8fff-4401-a991-1a661fd9c6e5\") " pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:30:39.759268 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:39.759251    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:39.824218 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:39.824190    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2prmp\""
Oct 30 19:30:39.831328 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:39.831303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:39.847860 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:39.847832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5z5s\" (UniqueName: \"kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s\") pod \"network-check-target-vwkv5\" (UID: \"101b32f8-ea50-4c3d-8114-15ebe26eb6d0\") " pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:30:39.848091 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:39.848074    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8hwq2\""
Oct 30 19:30:39.850159 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:39.850141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:30:39.851132 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:39.851115    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:30:39.861512 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:39.861494    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:30:39.872310 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:39.872287    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5z5s\" (UniqueName: \"kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s\") pod \"network-check-target-vwkv5\" (UID: \"101b32f8-ea50-4c3d-8114-15ebe26eb6d0\") " pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:30:40.110760 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:30:40.110722    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8013e357_0fa2_4360_924b_18cbfe588b5c.slice/crio-3889ae11c60344fc958acca1cee6adaec6562eb2ae83aa8b80d93ea893e7711b WatchSource:0}: Error finding container 3889ae11c60344fc958acca1cee6adaec6562eb2ae83aa8b80d93ea893e7711b: Status 404 returned error can't find the container with id 3889ae11c60344fc958acca1cee6adaec6562eb2ae83aa8b80d93ea893e7711b
Oct 30 19:30:40.111683 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:40.111657    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d2hv2"]
Oct 30 19:30:40.134503 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:40.134474    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-q9kwd\""
Oct 30 19:30:40.140575 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:40.140553    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mwcmw"]
Oct 30 19:30:40.142434 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:40.142405    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:30:40.143776 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:30:40.143753    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c979e36_8fff_4401_a991_1a661fd9c6e5.slice/crio-4ff348abb90c20e755aed86db9e1bc9fb9c6fce67ea6ac04d107527b657466b5 WatchSource:0}: Error finding container 4ff348abb90c20e755aed86db9e1bc9fb9c6fce67ea6ac04d107527b657466b5: Status 404 returned error can't find the container with id 4ff348abb90c20e755aed86db9e1bc9fb9c6fce67ea6ac04d107527b657466b5
Oct 30 19:30:40.415833 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:40.415764    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vwkv5"]
Oct 30 19:30:40.419161 ip-10-0-141-19 kubenswrapper[2577]: W1030 19:30:40.419134    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101b32f8_ea50_4c3d_8114_15ebe26eb6d0.slice/crio-b1e3a66ef45c22aacb12406e3c79ac8b4fab71f1a5e5ceea73fa4e81516fdcf7 WatchSource:0}: Error finding container b1e3a66ef45c22aacb12406e3c79ac8b4fab71f1a5e5ceea73fa4e81516fdcf7: Status 404 returned error can't find the container with id b1e3a66ef45c22aacb12406e3c79ac8b4fab71f1a5e5ceea73fa4e81516fdcf7
Oct 30 19:30:40.517634 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:40.517602    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vwkv5" event={"ID":"101b32f8-ea50-4c3d-8114-15ebe26eb6d0","Type":"ContainerStarted","Data":"b1e3a66ef45c22aacb12406e3c79ac8b4fab71f1a5e5ceea73fa4e81516fdcf7"}
Oct 30 19:30:40.518712 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:40.518685    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d2hv2" event={"ID":"8013e357-0fa2-4360-924b-18cbfe588b5c","Type":"ContainerStarted","Data":"3889ae11c60344fc958acca1cee6adaec6562eb2ae83aa8b80d93ea893e7711b"}
Oct 30 19:30:40.519719 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:40.519698    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mwcmw" event={"ID":"7c979e36-8fff-4401-a991-1a661fd9c6e5","Type":"ContainerStarted","Data":"4ff348abb90c20e755aed86db9e1bc9fb9c6fce67ea6ac04d107527b657466b5"}
Oct 30 19:30:41.526597 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:41.526559    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mwcmw" event={"ID":"7c979e36-8fff-4401-a991-1a661fd9c6e5","Type":"ContainerStarted","Data":"eb571c56cc4eb8ccacf64558eff2d27ee4a7689268de7ce24ecb574391fc3252"}
Oct 30 19:30:41.527083 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:41.526625    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mwcmw" event={"ID":"7c979e36-8fff-4401-a991-1a661fd9c6e5","Type":"ContainerStarted","Data":"4d53f188feb81380511d69251d48c30ffaf4d82164e4bf716e75cbdcb492cf89"}
Oct 30 19:30:41.549424 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:41.549380    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mwcmw" podStartSLOduration=66.477220746 podStartE2EDuration="1m7.549365617s" totalImagesPullingTime="1.072144871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:41.547322795 +0000 UTC m=+67.961961781" watchObservedRunningTime="2026-10-30 19:30:41.549365617 +0000 UTC m=+67.964004599"
Oct 30 19:30:43.293632 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:43.293594    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:43.294097 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:43.293687    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:43.299593 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:43.299570    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:43.539556 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:43.539524    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:43.597333 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:43.597231    2577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-854b9d69db-nvllp"]
Oct 30 19:30:44.539644 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:44.539607    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vwkv5" event={"ID":"101b32f8-ea50-4c3d-8114-15ebe26eb6d0","Type":"ContainerStarted","Data":"3df91e104211254ad0eeaa8ac34f317f98a75fe6c71ed3ac99848162719179f7"}
Oct 30 19:30:44.560168 ip-10-0-141-19 kubenswrapper[2577]: I1030 19:30:44.560113    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vwkv5" podStartSLOduration=67.207746054 podStartE2EDuration="1m10.560077453s" totalImagesPullingTime="3.352331399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:44.558893511 +0000 UTC m=+70.973532492" watchObservedRunningTime="2026-10-30 19:30:44.560077453 +0000 UTC m=+70.974716436"
Oct 30 19:30:45.352275 ip-10-0-141-19 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 19:30:45.372473 ip-10-0-141-19 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 19:30:45.372644 ip-10-0-141-19 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 19:30:45.372696 ip-10-0-141-19 systemd[1]: kubelet.service: Consumed 3.746s CPU time.
Oct 30 19:30:45.384532 ip-10-0-141-19 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:30:45.446757 ip-10-0-141-19 kubenswrapper[5408]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:30:45.446757 ip-10-0-141-19 kubenswrapper[5408]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:30:45.446757 ip-10-0-141-19 kubenswrapper[5408]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:30:45.446757 ip-10-0-141-19 kubenswrapper[5408]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:30:45.449183 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449086    5408 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:45.449183 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449178    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:45.449183 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449183    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:45.449183 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449188    5408 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449193    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449198    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449202    5408 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449205    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449209    5408 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449213    5408 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449216    5408 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449220    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449224    5408 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449228    5408 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449245    5408 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449249    5408 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449253    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449257    5408 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449261    5408 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449265    5408 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449270    5408 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449274    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449278    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:45.449456 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449282    5408 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449286    5408 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449291    5408 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449295    5408 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449300    5408 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449311    5408 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449315    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449319    5408 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449323    5408 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449327    5408 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449332    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449336    5408 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449343    5408 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449352    5408 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449358    5408 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449363    5408 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449367    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449372    5408 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449376    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:45.450334 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449381    5408 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449385    5408 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449390    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449394    5408 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449398    5408 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449403    5408 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449408    5408 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449412    5408 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449416    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449421    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449426    5408 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449430    5408 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449434    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449439    5408 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449443    5408 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449447    5408 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449451    5408 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449455    5408 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449460    5408 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449464    5408 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:45.451255 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449468    5408 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449473    5408 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449477    5408 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449482    5408 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449488    5408 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449493    5408 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449497    5408 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449502    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449506    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449511    5408 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449515    5408 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449520    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449524    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449528    5408 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449533    5408 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449537    5408 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449541    5408 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449546    5408 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449552    5408 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449556    5408 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:45.452160 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449560    5408 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449565    5408 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449569    5408 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449573    5408 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449577    5408 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449582    5408 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449587    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449591    5408 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449598    5408 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449606    5408 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449611    5408 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449616    5408 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449622    5408 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449626    5408 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449630    5408 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449635    5408 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449640    5408 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449644    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449648    5408 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:45.453075 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449653    5408 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449657    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449661    5408 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449667    5408 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.449671    5408 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.450938    5408 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.450949    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.450953    5408 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.450958    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.450963    5408 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.450967    5408 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.450971    5408 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.450984    5408 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.450993    5408 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.450999    5408 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451004    5408 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451008    5408 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451012    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451017    5408 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451021    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:45.453945 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451045    5408 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451051    5408 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451055    5408 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451059    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451064    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451068    5408 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451072    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451077    5408 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451081    5408 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451085    5408 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451090    5408 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451095    5408 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451100    5408 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451104    5408 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451109    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451113    5408 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451117    5408 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451121    5408 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451125    5408 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:45.454850 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451130    5408 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451134    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451138    5408 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451143    5408 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451148    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451152    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451165    5408 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451169    5408 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451173    5408 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451178    5408 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451183    5408 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451187    5408 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451191    5408 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451196    5408 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451200    5408 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451204    5408 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451208    5408 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451214    5408 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451222    5408 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451227    5408 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:45.455657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451231    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451235    5408 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451241    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451245    5408 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451250    5408 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451254    5408 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451258    5408 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451262    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451266    5408 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451270    5408 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451274    5408 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451278    5408 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451282    5408 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451289    5408 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451294    5408 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451299    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451304    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451309    5408 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451313    5408 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:45.456227 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451318    5408 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451322    5408 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451327    5408 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451332    5408 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451336    5408 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451340    5408 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451345    5408 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451349    5408 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451354    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451358    5408 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451363    5408 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451369    5408 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451373    5408 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451380    5408 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451384    5408 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451389    5408 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451393    5408 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451397    5408 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451402    5408 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451406    5408 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:45.456707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451410    5408 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451414    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451418    5408 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451422    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451427    5408 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451431    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451436    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451440    5408 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451444    5408 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451449    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451454    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451458    5408 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.451462    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452325    5408 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452335    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452341    5408 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452345    5408 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452352    5408 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452358    5408 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:45.457230 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452362    5408 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452367    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452372    5408 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452377    5408 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452382    5408 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452388    5408 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452392    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452396    5408 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452401    5408 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452405    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452409    5408 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452414    5408 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452418    5408 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452423    5408 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452427    5408 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452433    5408 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452437    5408 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452442    5408 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452446    5408 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452450    5408 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:45.457813 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452455    5408 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452459    5408 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452463    5408 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452468    5408 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452473    5408 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452477    5408 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452482    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452486    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452490    5408 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452495    5408 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452499    5408 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452503    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452507    5408 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452512    5408 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452517    5408 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452521    5408 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452525    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452530    5408 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452534    5408 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452538    5408 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:45.458324 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452542    5408 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452547    5408 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452550    5408 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452555    5408 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452560    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452564    5408 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452569    5408 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452573    5408 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452577    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452582    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452586    5408 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452590    5408 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452594    5408 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452599    5408 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452603    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452607    5408 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452614    5408 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452618    5408 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452623    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452628    5408 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:45.458814 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452632    5408 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452637    5408 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452643    5408 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452650    5408 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452655    5408 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452660    5408 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452666    5408 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452671    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452675    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452679    5408 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452684    5408 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452688    5408 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452692    5408 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452697    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452702    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452706    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452710    5408 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452715    5408 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452719    5408 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:45.459389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452723    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452727    5408 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452732    5408 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452736    5408 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452741    5408 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452745    5408 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452749    5408 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452754    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452758    5408 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452764    5408 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452768    5408 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452773    5408 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452777    5408 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452782    5408 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452787    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452791    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452796    5408 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452800    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452805    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452810    5408 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:45.459849 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.452814    5408 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.452959    5408 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.452970    5408 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.452979    5408 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.452987    5408 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.452994    5408 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.452999    5408 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453006    5408 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453012    5408 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453017    5408 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453022    5408 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453048    5408 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453054    5408 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453059    5408 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453063    5408 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453068    5408 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453073    5408 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453077    5408 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453082    5408 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453089    5408 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453093    5408 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453101    5408 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453106    5408 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453112    5408 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:30:45.460680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453118    5408 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453123    5408 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453131    5408 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453137    5408 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453142    5408 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453147    5408 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453152    5408 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453157    5408 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453162    5408 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453169    5408 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453175    5408 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453180    5408 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453185    5408 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453190    5408 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453195    5408 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453202    5408 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453207    5408 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453212    5408 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453217    5408 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453223    5408 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453230    5408 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453234    5408 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453240    5408 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453245    5408 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453250    5408 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:30:45.461556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453254    5408 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453259    5408 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453264    5408 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453269    5408 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453276    5408 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453281    5408 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453288    5408 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453293    5408 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453298    5408 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453313    5408 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453319    5408 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453324    5408 flags.go:64] FLAG: --help="false"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453329    5408 flags.go:64] FLAG: --hostname-override="ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453335    5408 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453339    5408 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453344    5408 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453351    5408 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453357    5408 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453362    5408 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453366    5408 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453371    5408 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453376    5408 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453381    5408 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:30:45.462192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453385    5408 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453390    5408 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453395    5408 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453400    5408 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453405    5408 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453410    5408 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453415    5408 flags.go:64] FLAG: --lock-file=""
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453419    5408 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453424    5408 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453429    5408 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453439    5408 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453444    5408 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453449    5408 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453456    5408 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453460    5408 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453466    5408 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453471    5408 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453475    5408 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453485    5408 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453490    5408 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453497    5408 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453502    5408 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453507    5408 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453512    5408 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453517    5408 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:30:45.462752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453522    5408 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453527    5408 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453532    5408 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453543    5408 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453548    5408 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453553    5408 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453558    5408 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453563    5408 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453568    5408 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453573    5408 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453578    5408 flags.go:64] FLAG: --port="10250"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453583    5408 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453587    5408 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-09ea4d04e90b55a23"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453593    5408 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453598    5408 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453602    5408 flags.go:64] FLAG: --register-node="true"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453607    5408 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453618    5408 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453623    5408 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453629    5408 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453635    5408 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453642    5408 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453646    5408 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453652    5408 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453657    5408 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453663    5408 flags.go:64] FLAG: --runonce="false"
Oct 30 19:30:45.463369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453668    5408 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453673    5408 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453679    5408 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453684    5408 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453689    5408 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453694    5408 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453699    5408 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453704    5408 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453709    5408 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453713    5408 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453718    5408 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453723    5408 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453728    5408 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453733    5408 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453738    5408 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453747    5408 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453751    5408 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453756    5408 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453767    5408 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453771    5408 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453777    5408 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453781    5408 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453786    5408 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453791    5408 flags.go:64] FLAG: --v="2"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453798    5408 flags.go:64] FLAG: --version="false"
Oct 30 19:30:45.464003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453805    5408 flags.go:64] FLAG: --vmodule=""
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453814    5408 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.453819    5408 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454061    5408 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454068    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454073    5408 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454081    5408 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454086    5408 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454090    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454095    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454101    5408 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454105    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454110    5408 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454114    5408 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454118    5408 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454123    5408 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454127    5408 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454132    5408 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454136    5408 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454141    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:45.464677 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454145    5408 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454149    5408 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454154    5408 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454157    5408 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454161    5408 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454166    5408 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454171    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454175    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454179    5408 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454184    5408 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454191    5408 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454198    5408 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454203    5408 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454211    5408 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454216    5408 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454220    5408 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454225    5408 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454230    5408 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454236    5408 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:45.465257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454240    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454244    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454249    5408 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454254    5408 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454259    5408 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454263    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454268    5408 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454272    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454277    5408 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454281    5408 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454285    5408 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454290    5408 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454294    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454299    5408 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454303    5408 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454308    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454312    5408 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454317    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454322    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:45.465951 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454328    5408 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454335    5408 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454340    5408 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454345    5408 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454351    5408 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454356    5408 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454360    5408 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454367    5408 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454371    5408 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454376    5408 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454380    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454385    5408 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454391    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454396    5408 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454400    5408 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454405    5408 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454409    5408 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454414    5408 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454419    5408 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454424    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:45.466526 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454428    5408 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454433    5408 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454437    5408 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454441    5408 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454445    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454450    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454454    5408 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454458    5408 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454463    5408 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454468    5408 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454472    5408 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454476    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454481    5408 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454485    5408 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454490    5408 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454494    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454498    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454503    5408 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454508    5408 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:45.467023 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454514    5408 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:45.467761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454518    5408 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:45.467761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454523    5408 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:45.467761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454527    5408 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:45.467761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454532    5408 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:45.467761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454538    5408 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:45.467761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454543    5408 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:45.467761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454547    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:45.467761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454551    5408 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:45.467761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454556    5408 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:45.467761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454560    5408 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:45.467761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454564    5408 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:45.467761 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.454609    5408 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:30:45.467761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454666    5408 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:45.467761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454674    5408 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:45.467761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454678    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:45.467761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454683    5408 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454689    5408 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454694    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454698    5408 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454702    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454707    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454714    5408 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454719    5408 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454724    5408 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454728    5408 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454733    5408 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454737    5408 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454742    5408 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454746    5408 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454750    5408 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454754    5408 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454760    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454765    5408 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454769    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:45.468260 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454774    5408 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454779    5408 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454783    5408 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454789    5408 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454793    5408 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454797    5408 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454802    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454806    5408 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454810    5408 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454815    5408 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454820    5408 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454825    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454829    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454833    5408 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454838    5408 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454842    5408 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454846    5408 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454850    5408 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454854    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:45.468775 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454858    5408 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454862    5408 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454866    5408 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454870    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454875    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454879    5408 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454884    5408 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454888    5408 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454892    5408 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454897    5408 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454903    5408 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454908    5408 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454912    5408 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454916    5408 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454920    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454925    5408 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454930    5408 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454935    5408 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454939    5408 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454944    5408 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:45.469254 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454948    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454953    5408 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454958    5408 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454963    5408 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454967    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454973    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454978    5408 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454982    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454986    5408 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454991    5408 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454995    5408 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.454999    5408 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455003    5408 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455008    5408 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455012    5408 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455017    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455021    5408 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455041    5408 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455046    5408 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455050    5408 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:45.469747 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455055    5408 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455059    5408 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455065    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455070    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455074    5408 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455078    5408 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455085    5408 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455091    5408 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455098    5408 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455104    5408 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455108    5408 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455113    5408 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455119    5408 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455123    5408 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455128    5408 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455132    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455136    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455142    5408 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455146    5408 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:45.470261 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455150    5408 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455155    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455159    5408 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455163    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455167    5408 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.455171    5408 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.459007    5408 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.459023    5408 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.459056    5408 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.459061    5408 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459116    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459122    5408 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459125    5408 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459129    5408 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459132    5408 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459135    5408 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459138    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459141    5408 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459144    5408 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:45.470748 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459147    5408 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459150    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459153    5408 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459155    5408 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459158    5408 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459161    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459165    5408 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459169    5408 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459172    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459174    5408 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459177    5408 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459179    5408 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459182    5408 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459185    5408 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459187    5408 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459190    5408 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459193    5408 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459196    5408 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459199    5408 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:45.471310 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459202    5408 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459205    5408 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459208    5408 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459211    5408 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459214    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459217    5408 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459220    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459222    5408 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459225    5408 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459228    5408 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459231    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459234    5408 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459236    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459239    5408 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459242    5408 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459245    5408 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459248    5408 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459251    5408 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459254    5408 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459256    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:45.471763 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459259    5408 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459262    5408 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459265    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459267    5408 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459270    5408 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459273    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459275    5408 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459278    5408 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459280    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459283    5408 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459285    5408 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459289    5408 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459293    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459297    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459301    5408 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459303    5408 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459306    5408 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459309    5408 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459311    5408 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:45.472267 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459314    5408 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459317    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459319    5408 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459322    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459325    5408 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459328    5408 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459331    5408 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459333    5408 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459337    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459339    5408 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459342    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459345    5408 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459347    5408 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459350    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459352    5408 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459355    5408 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459358    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459360    5408 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459363    5408 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:45.472761 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459365    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459368    5408 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459370    5408 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459373    5408 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459376    5408 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459378    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459381    5408 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459384    5408 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459387    5408 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459390    5408 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459393    5408 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459396    5408 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459399    5408 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459402    5408 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459405    5408 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459407    5408 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459410    5408 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459413    5408 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459415    5408 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:45.473269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459418    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:45.473873 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.459444    5408 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:30:45.473873 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459579    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:45.473873 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459583    5408 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:45.473873 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459586    5408 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:45.473873 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459589    5408 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:45.473873 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459591    5408 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:45.473873 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459594    5408 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:45.473873 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459597    5408 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:45.473873 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459599    5408 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:45.473873 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459602    5408 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:45.473873 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459605    5408 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:45.473873 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459608    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:45.473873 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459610    5408 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:45.473873 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459613    5408 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:45.473873 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459615    5408 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:45.473873 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459618    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459621    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459623    5408 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459626    5408 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459629    5408 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459631    5408 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459634    5408 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459637    5408 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459640    5408 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459642    5408 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459645    5408 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459648    5408 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459650    5408 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459653    5408 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459656    5408 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459658    5408 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459661    5408 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459663    5408 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459666    5408 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459668    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:45.474387 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459671    5408 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459673    5408 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459676    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459679    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459681    5408 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459684    5408 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459686    5408 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459689    5408 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459692    5408 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459694    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459698    5408 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459701    5408 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459705    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459707    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459710    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459713    5408 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459716    5408 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459718    5408 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459721    5408 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:45.474973 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459723    5408 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459727    5408 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459730    5408 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459733    5408 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459736    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459738    5408 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459741    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459743    5408 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459746    5408 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459748    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459751    5408 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459753    5408 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459756    5408 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459758    5408 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459761    5408 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459764    5408 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459766    5408 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459769    5408 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459771    5408 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459774    5408 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:45.475596 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459776    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459779    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459782    5408 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459784    5408 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459787    5408 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459789    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459792    5408 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459794    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459797    5408 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459799    5408 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459802    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459805    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459807    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459810    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459813    5408 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459816    5408 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459819    5408 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459822    5408 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459824    5408 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:45.476259 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459827    5408 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:45.476811 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459829    5408 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:45.476811 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459832    5408 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:45.476811 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459835    5408 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:45.476811 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459837    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:45.476811 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459840    5408 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:45.476811 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459843    5408 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:45.476811 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459846    5408 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:45.476811 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459850    5408 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:45.476811 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459853    5408 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:45.476811 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459856    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:45.476811 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459859    5408 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:45.476811 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459862    5408 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:45.476811 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459864    5408 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:45.476811 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.459887    5408 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:30:45.476811 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459917    5408 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:45.476811 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459922    5408 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459926    5408 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459929    5408 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459931    5408 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459934    5408 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459937    5408 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459939    5408 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459942    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459944    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459947    5408 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459950    5408 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459952    5408 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459955    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459958    5408 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459961    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459963    5408 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459966    5408 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459968    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459971    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459974    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:45.477223 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459976    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459979    5408 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459982    5408 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459984    5408 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459987    5408 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459989    5408 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459992    5408 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459994    5408 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.459997    5408 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460000    5408 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460002    5408 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460005    5408 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460008    5408 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460012    5408 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460016    5408 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460019    5408 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460021    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460024    5408 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460038    5408 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:45.477736 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460041    5408 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460043    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460046    5408 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460048    5408 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460051    5408 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460054    5408 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460105    5408 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460282    5408 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460292    5408 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460298    5408 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460303    5408 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460307    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460312    5408 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460317    5408 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460321    5408 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460326    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460331    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460335    5408 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460340    5408 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460345    5408 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:45.478265 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460355    5408 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460359    5408 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460364    5408 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460371    5408 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460376    5408 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460380    5408 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460384    5408 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460388    5408 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460393    5408 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460398    5408 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460402    5408 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460406    5408 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460417    5408 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460422    5408 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460426    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460431    5408 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460435    5408 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460440    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460479    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460497    5408 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:45.478997 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460503    5408 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460508    5408 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460513    5408 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460518    5408 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460527    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460534    5408 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460540    5408 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460545    5408 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460549    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460554    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460558    5408 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460563    5408 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460567    5408 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460571    5408 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460576    5408 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460581    5408 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460585    5408 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460595    5408 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460599    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:45.479659 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460604    5408 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:45.480401 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460608    5408 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:45.480401 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460612    5408 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:45.480401 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460617    5408 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:45.480401 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460621    5408 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:45.480401 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460625    5408 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:45.480401 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:45.460632    5408 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:45.480401 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.460750    5408 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:30:45.480401 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.462412    5408 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 19:30:45.480401 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.462467    5408 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 19:30:45.480401 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.462649    5408 server.go:1014] "Starting client certificate rotation"
Oct 30 19:30:45.480401 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.462783    5408 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:30:45.480401 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.462836    5408 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:24:33 +0000 UTC" deadline="2028-05-28 07:04:36.846634406 +0000 UTC"
Oct 30 19:30:45.480401 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.462872    5408 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13811h33m51.383764666s"
Oct 30 19:30:45.480401 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.463164    5408 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:30:45.480401 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.464235    5408 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:30:45.480401 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.465474    5408 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:30:45.481019 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.467347    5408 log.go:25] "Validated CRI v1 image API"
Oct 30 19:30:45.481019 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.468103    5408 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:30:45.481019 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.470087    5408 fs.go:135] Filesystem UUIDs: map[1f5c32e2-90df-4269-bedf-fadc30face1e:/dev/nvme0n1p3 31d6c66c-1be2-4143-969e-87a199824e2e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 19:30:45.481167 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.470102    5408 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f3a89bbe93fdc66236bd2765050de4d26d191fef04c0713b85c04d3b08a37fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f3a89bbe93fdc66236bd2765050de4d26d191fef04c0713b85c04d3b08a37fc/userdata/shm major:0 minor:327 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3889ae11c60344fc958acca1cee6adaec6562eb2ae83aa8b80d93ea893e7711b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3889ae11c60344fc958acca1cee6adaec6562eb2ae83aa8b80d93ea893e7711b/userdata/shm major:0 minor:367 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e80dc474dd09ed7516ce7158907a43843349e37465b9e0ab5ff913cfce5c9ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e80dc474dd09ed7516ce7158907a43843349e37465b9e0ab5ff913cfce5c9ce/userdata/shm major:0 minor:302 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47ee8ad7afa297228f1b3adfeb887e1799deacaaad1a2cabc351691e1f0a39c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47ee8ad7afa297228f1b3adfeb887e1799deacaaad1a2cabc351691e1f0a39c5/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4cda2bf1ae154e1b02cf339cbcb9249ff3eb3aa791a8213ba323612fa10282cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4cda2bf1ae154e1b02cf339cbcb9249ff3eb3aa791a8213ba323612fa10282cc/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ff348abb90c20e755aed86db9e1bc9fb9c6fce67ea6ac04d107527b657466b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ff348abb90c20e755aed86db9e1bc9fb9c6fce67ea6ac04d107527b657466b5/userdata/shm major:0 minor:369 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/771d5ecbcb285260e210781715caeda168dd97a9820a0dfb0cb9a687f8a1c7ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/771d5ecbcb285260e210781715caeda168dd97a9820a0dfb0cb9a687f8a1c7ec/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/823e2f643e9b2f48bd55729c3f8353509829c76f10b6e67696ff3b2964280b0f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/823e2f643e9b2f48bd55729c3f8353509829c76f10b6e67696ff3b2964280b0f/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90341f35811b436dab6f5eeb57758a8f6c6c901b850325b0566b18661bd1a6ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90341f35811b436dab6f5eeb57758a8f6c6c901b850325b0566b18661bd1a6ed/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/905a4f91743ddd1fb3ee8d85a3d61109230e811ed1b2c825e5e05dae8ccd8b7b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/905a4f91743ddd1fb3ee8d85a3d61109230e811ed1b2c825e5e05dae8ccd8b7b/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1e3a66ef45c22aacb12406e3c79ac8b4fab71f1a5e5ceea73fa4e81516fdcf7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1e3a66ef45c22aacb12406e3c79ac8b4fab71f1a5e5ceea73fa4e81516fdcf7/userdata/shm major:0 minor:376 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c326058e8f2e9a4ae0387a68eab1d87ddbdf11405269f074395b5f275417f1c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c326058e8f2e9a4ae0387a68eab1d87ddbdf11405269f074395b5f275417f1c6/userdata/shm major:0 minor:355 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3e72d26b223d07b1cab9286bc95e64e8f5540868af1598407589ef50ee834aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3e72d26b223d07b1cab9286bc95e64e8f5540868af1598407589ef50ee834aa/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c40aa314795318f3f4dd9529bcfa37edcc6c76c12978d657b607d6cd3ae69209/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c40aa314795318f3f4dd9529bcfa37edcc6c76c12978d657b607d6cd3ae69209/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6385dafaa9f8c8e977f2d44f2491d84e9755c9c1ae80b00e04f2ca29bcdbf8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6385dafaa9f8c8e977f2d44f2491d84e9755c9c1ae80b00e04f2ca29bcdbf8b/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfd93681f5f218e22c883a9e49151eb7f586674f01d276757174f9829a421c37/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfd93681f5f218e22c883a9e49151eb7f586674f01d276757174f9829a421c37/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e30bb07f1caa152986c84a26125c0e5346b0e8bc552b5156f435783366de1ce9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e30bb07f1caa152986c84a26125c0e5346b0e8bc552b5156f435783366de1ce9/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e59da0f23b582ab57c0b73f7043466ba70b4e7e2b61094e81723590699ea17f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e59da0f23b582ab57c0b73f7043466ba70b4e7e2b61094e81723590699ea17f1/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed3d450f2f9f551f3fe9791bd70422e1170a01b7f13bf5a069d6f897d8939011/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed3d450f2f9f551f3fe9791bd70422e1170a01b7f13bf5a069d6f897d8939011/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4174a924aa6c72d9f7eeab913165b762220275baf3d845126dc6629a256070b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4174a924aa6c72d9f7eeab913165b762220275baf3d845126dc6629a256070b/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4d92d32030cb453da5f69397058a394d5892d1066c08079fb94644e86839df7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4d92d32030cb453da5f69397058a394d5892d1066c08079fb94644e86839df7/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5b879a0b0afdc510709a56fedddbe3e63eee4eb9a2d869df77a7d9f3235cde6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5b879a0b0afdc510709a56fedddbe3e63eee4eb9a2d869df77a7d9f3235cde6/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/008928f8-b2ce-4f12-9681-21136d802ff6/volumes/kubernetes.io~projected/kube-api-access-j8bxp:{mountpoint:/var/lib/kubelet/pods/008928f8-b2ce-4f12-9681-21136d802ff6/volumes/kubernetes.io~projected/kube-api-access-j8bxp major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/008928f8-b2ce-4f12-9681-21136d802ff6/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/008928f8-b2ce-4f12-9681-21136d802ff6/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/008928f8-b2ce-4f12-9681-21136d802ff6/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/008928f8-b2ce-4f12-9681-21136d802ff6/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0de2915a-789c-4acc-84a6-c61987498b62/volumes/kubernetes.io~projected/kube-api-access-nxxqw:{mountpoint:/var/lib/kubelet/pods/0de2915a-789c-4acc-84a6-c61987498b62/volumes/kubernetes.io~projected/kube-api-access-nxxqw major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0de2915a-789c-4acc-84a6-c61987498b62/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0de2915a-789c-4acc-84a6-c61987498b62/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf/volumes/kubernetes.io~projected/kube-api-access-7z7pj:{mountpoint:/var/lib/kubelet/pods/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf/volumes/kubernetes.io~projected/kube-api-access-7z7pj major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/101b32f8-ea50-4c3d-8114-15ebe26eb6d0/volumes/kubernetes.io~projected/kube-api-access-d5z5s:{mountpoint:/var/lib/kubelet/pods/101b32f8-ea50-4c3d-8114-15ebe26eb6d0/volumes/kubernetes.io~projected/kube-api-access-d5z5s major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23de5b1c-4825-43ed-bff8-cdab8789989e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/23de5b1c-4825-43ed-bff8-cdab8789989e/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e9972f4-8918-41a0-8ae4-b2014d3897ca/volumes/kubernetes.io~projected/kube-api-access-b4btm:{mountpoint:/var/lib/kubelet/pods/3e9972f4-8918-41a0-8ae4-b2014d3897ca/volumes/kubernetes.io~projected/kube-api-access-b4btm major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e9972f4-8918-41a0-8ae4-b2014d3897ca/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/3e9972f4-8918-41a0-8ae4-b2014d3897ca/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e9972f4-8918-41a0-8ae4-b2014d3897ca/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/3e9972f4-8918-41a0-8ae4-b2014d3897ca/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45ccae2d-1b91-4bbe-9598-94ad740116ec/volumes/kubernetes.io~projected/kube-api-access-d5sb5:{mountpoint:/var/lib/kubelet/pods/45ccae2d-1b91-4bbe-9598-94ad740116ec/volumes/kubernetes.io~projected/kube-api-access-d5sb5 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45ccae2d-1b91-4bbe-9598-94ad740116ec/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/45ccae2d-1b91-4bbe-9598-94ad740116ec/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53bd98ea-ea87-484a-b7b1-8c9070fc9b25/volumes/kubernetes.io~projected/kube-api-access-b9tds:{mountpoint:/var/lib/kubelet/pods/53bd98ea-ea87-484a-b7b1-8c9070fc9b25/volumes/kubernetes.io~projected/kube-api-access-b9tds major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68ff550a-daae-421b-9dd6-e9b1f33585d1/volumes/kubernetes.io~projected/kube-api-access-9tcgh:{mountpoint:/var/lib/kubelet/pods/68ff550a-daae-421b-9dd6-e9b1f33585d1/volumes/kubernetes.io~projected/kube-api-access-9tcgh major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c979e36-8fff-4401-a991-1a661fd9c6e5/volumes/kubernetes.io~projected/kube-api-access-csj5l:{mountpoint:/var/lib/kubelet/pods/7c979e36-8fff-4401-a991-1a661fd9c6e5/volumes/kubernetes.io~projected/kube-api-access-csj5l major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c979e36-8fff-4401-a991-1a661fd9c6e5/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/7c979e36-8fff-4401-a991-1a661fd9c6e5/volumes/kubernetes.io~secret/metrics-certs major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8013e357-0fa2-4360-924b-18cbfe588b5c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8013e357-0fa2-4360-924b-18cbfe588b5c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57/volumes/kubernetes.io~projected/kube-api-access-k9fpq:{mountpoint:/var/lib/kubelet/pods/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57/volumes/kubernetes.io~projected/kube-api-access-k9fpq major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a341b7ef-207b-4443-b01e-9fcbc2ac0075/volumes/kubernetes.io~projected/kube-api-access-d895r:{mountpoint:/var/lib/kubelet/pods/a341b7ef-207b-4443-b01e-9fcbc2ac0075/volumes/kubernetes.io~projected/kube-api-access-d895r major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a341b7ef-207b-4443-b01e-9fcbc2ac0075/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/a341b7ef-207b-4443-b01e-9fcbc2ac0075/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c112dbe2-b2e4-4762-8123-b9f161bda84a/volumes/kubernetes.io~projected/kube-api-access-6k7xh:{mountpoint:/var/lib/kubelet/pods/c112dbe2-b2e4-4762-8123-b9f161bda84a/volumes/kubernetes.io~projected/kube-api-access-6k7xh major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c112dbe2-b2e4-4762-8123-b9f161bda84a/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/c112dbe2-b2e4-4762-8123-b9f161bda84a/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c112dbe2-b2e4-4762-8123-b9f161bda84a/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/c112dbe2-b2e4-4762-8123-b9f161bda84a/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2836676-0428-4f2f-a801-0a25df87aee2/volumes/kubernetes.io~projected/kube-api-access-xbkb8:{mountpoint:/var/lib/kubelet/pods/c2836676-0428-4f2f-a801-0a25df87aee2/volumes/kubernetes.io~projected/kube-api-access-xbkb8 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5/volumes/kubernetes.io~projected/kube-api-access-q5sd9:{mountpoint:/var/lib/kubelet/pods/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5/volumes/kubernetes.io~projected/kube-api-access-q5sd9 major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d50ec904-b3c9-4677-9624-39cb202f65c9/volumes/kubernetes.io~projected/kube-api-access-pmjjp:{mountpoint:/var/lib/kubelet/pods/d50ec904-b3c9-4677-9624-39cb202f65c9/volumes/kubernetes.io~projected/kube-api-access-pmjjp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de8ddc3f-5d23-4cff-9f2e-9167700ff109/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/de8ddc3f-5d23-4cff-9f2e-9167700ff109/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de8ddc3f-5d23-4cff-9f2e-9167700ff109/volumes/kubernetes.io~projected/kube-api-access-f6hdz:{mountpoint:/var/lib/kubelet/pods/de8ddc3f-5d23-4cff-9f2e-9167700ff109/volumes/kubernetes.io~projected/kube-api-access-f6hdz major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de8ddc3f-5d23-4cff-9f2e-9167700ff109/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/de8ddc3f-5d23-4cff-9f2e-9167700ff109/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0a98773-9cb1-4637-9188-52094d47dceb/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e0a98773-9cb1-4637-9188-52094d47dceb/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0a98773-9cb1-4637-9188-52094d47dceb/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e0a98773-9cb1-4637-9188-52094d47dceb/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0a98773-9cb1-4637-9188-52094d47dceb/volumes/kubernetes.io~projected/kube-api-access-d6v4c:{mountpoint:/var/lib/kubelet/pods/e0a98773-9cb1-4637-9188-52094d47dceb/volumes/kubernetes.io~projected/kube-api-access-d6v4c major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~projected/kube-api-access-w5rj9:{mountpoint:/var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~projected/kube-api-access-w5rj9 major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2ac53be-96b4-4976-9581-b38cc97e1eb0/volumes/kubernetes.io~projected/kube-api-access-l6scd:{mountpoint:/var/lib/kubelet/pods/e2ac53be-96b4-4976-9581-b38cc97e1eb0/volumes/kubernetes.io~projected/kube-api-access-l6scd major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/101e886e80de180480a26db07909c65da1ae864159efa14137e1bde50a8dc9ea/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6a872750bcb32e92b826ea385d706d9b872cd98c4e3753efe428c1b82d6b6671/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/29630d57e5bc7c2385f7c728e1d8ead18d04ea29f93f57b8ba6709edf31b85a4/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/4a325bda5b9d235eef8e21d0e81ae4eccacf2bbd2c1e3b3d8b513710ebc8f86a/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/96f3242805ac8cb3e9cf73ad1a2286cb5f37bfc05161f4a38867fe667be3def9/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/fa7805f99a8909c026a72388560e698b0c0c5d3c3e61b6ea62f55d375972cae1/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/31d5a079f575eaa0684945545380ca85cb5e2a5895ad4db22b3d5341b4f56b8d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/424d6faeda5935e85162016513daf66d9122f4a65355f51310935f95826bdf47/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/0def8e279a111dc7d46927e0ec586791d83ccbbd232b43072826bb7dae4ad892/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/6ce1cc759d7ba2cc5d6b313d8f73e7a5bacbe1c88fb0abb609ca8bd2a778106e/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/0c90c037fc81d3490a1ee678d4b502a0d58f5ddd7a0a47dabec6f976f9cec552/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/c0242cdd0e72bdc0fe49f93a0af541d74c83f7fe77ec5b018c51dfd1374a5e9e/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e8fd5ec16dd80eb5e8a86693e3ed89caaace56e23a9797d71f8f1dd477e2143c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/e0b6510430a6cf89022779fe0acc0acff643cb89b5b2a49d34d53fec76dba344/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d3d62aab9ac2a1c9ea9d2099ae0bd8abcb0667fa3ba4f8531645ac6014540bd7/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8226c0e724a468f8b2c1c53e8d0d492aec541e7424d6723230ade5575ead256b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/3797d5509cb89795ac37ecc148bd4b9b5871107e961cbe93f5d1085846b1610e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/7ebf19fef987f90a10c9ef1ed2b2dd78ee81d4952548717ddecb97df3b57158a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/1409278e9628352a6f85207ea628aad16c7f6ea3d54107e29be46dc4dd038356/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/9b284b9fe8d7838520fd2db55a74839019b56c2d55c74ea1cdad173e3a2cf685/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d001dc296b2c9e7a75cd01ed415efc892f3eada41e45e26daac85f7ac9fd46ee/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/f11bce765aef0073dded732971d78c1a3c5de89ce3592cd38477bafe41b02052/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/c6ada0ae78d199b484467f5833c399c497756593af74e3ed6be785a41b1f944c/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/18739dd3199d5729d72d07e0ba0ed08948727ac1d8c86dfbe2303f9574f1cb64/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/1c98202d4c19b75d36bdd0216d5bfce190cf8c5314178b86bf3759d13466413a/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/827a88976015039f55064ee81a6e7196f52ec73ba9fc868800979d02132131ad/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/4131b5dbecf484292e11cd267116310ab6e0e7f0f3358d2c7defa42b9d4771cd/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/0f9c92d21f6de9fed9c0df630b7ba11b1faed66ed41def09398f7ed2c22dde9a/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/8f379ec202cddba061bc3596b85567ab9108ce6d3d30408274bde3ca2043da48/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/dd301800f6ff3e1928305e31c36ffd908a0d87b283d0c81d51d2543fca67612d/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/faafabdb867fb86ed273b33ddbcba0f567d0fe88e98bf899da7272292dabe3c3/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/8a2353bc236927b883267fc49b06362362212436b95a128bbf3a04fd8c5588bc/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/bb74c990d5dc147cee3474b1e8c737a5dfda7183d3f0bb60d9a9f4ed5e55eb4f/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/42808174cec63b21279a5126f066262784588e5dee6f476fb96e8e197089dd77/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/58b90a88c278f88f0a0ea7f3be2267c05874a7cea42698bfd953e6e68bc0523f/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/cc060c08ebf505ac72a87b3b922ed1546d5b27cce693740ecdfeabfe87426178/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/640a28dfb3e5e07e30d62417d85aebf975cc07d64db0ee3d3e7e22e58d2d8c2b/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/147866dcd963ccf875813a50aa809c66f6cd1dcf421476ada32fd7aa24970674/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/ce5acb9a1e07d1d460341f8fb114952e8a6c8d81dd3ab25b656126da550cf109/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/8028a94ff3ee446c5381bd926937b79864bdf4ba3230946956be68b83bb25cf6/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/08c102474b4809072dd6aaad6117c33f2611559e52a831e1b2bc4881fb6de5e6/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/f72db64608bf0357f77e087443ea5d01683582443e15b1b55771fdf29e15824d/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/dc644a0be4f646e8fa3f4eabb2ff7faff9d9ab51876ffb8e510f29d6628a2d55/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/f3f8a2fd5eb6d216e589c7186971a2fa9807f0fab10baef7b4e567de58fd84bc/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/94cd3379855113198581cf765bc069d51d080a2a1b04e8a5efe8784d9da51686/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/b819c19dd9f178df435535b4f9afcfbd9eb7674c3483dcf5d08419efabf17677/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/64e420f2ed8c261db8aeb1a6c5a254fb4024d506f122e006a9900fec83d70837/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/4be61ab010edda34be8e40bff4c325f672e1a7856a15c05f70f5ab47f03fdaa8/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/3b04ce18d6b43ce6ffc8491e89bdbcc7e723197264b826285a7b45b33c8c8c38/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/5116df7f067c36850be8eb963f8ebf86ff33dc13b76971b952151890ac5f6187/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/3653053d9297c8009d570e77c486342c38841c5d5771f3b0a6f563a9991e929c/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/e4c2e29313f33045b52e45e0931878b00d58692919db869607839f48cbf23dca/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/f4d243250bd11e575c10302fae94080fc405a459501575c4599ce61194744424/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/4b312a82c43e0ced2d048d8f30c93ee327b681d22ed862507457f18bce8aab25/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/32fe017ef68a65f7b7e7100ccb47e579e5100fe89efc4ba054694f56ebdefddc/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/66e347d6b16878698c3debefc3312d74279b35a75f47bbfd37880ad82e99941f/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/58f212a303f0cc93278bdfdd2096d93f463724890fb6aca44a85a11d3a1dd5e6/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/cac6b8a2cfadfe784ee7e043a50bd302643d76113bf935baa165126c20d0a2dd/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/6e4865d6ae336eb36d1b63fed74fda6ac74a83795e9248ae183d7747393ee532/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c176771d15d5af201adf17bf322406b7f26c7d84006407daa084afdcfc56d110/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/b03b1b6f35498d8f35d7f273cd8a92159b78e4c1ec2db05e931d72d3ab94a376/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/5ae1334c736787db27adc4ae5dd89a5d6ea0e9ccf53f1747cf3603de7e4cf26b/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/8dc5673acd14e2528e3754afcffed168d736e083f6eed658f72f09513dcc98eb/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/3368a2a4c80558cfc80b400052e4da89b5d3efb2b23724bb65c45e839320f7ed/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/35cca82591129d1220981fb7b8aeed9f45a9468e99639388a185721efe9a81d2/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/becef729ecd093f272eb99deec49a290e23e25c01d7831f9c8a034d873b7d587/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.477158    5408 manager.go:217] Machine: {Timestamp:2026-10-30 19:30:45.476681035 +0000 UTC m=+0.060950645 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107267 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aaf5fc670c91356b37b6d6ac40f98 SystemUUID:ec2aaf5f-c670-c913-56b3-7b6d6ac40f98 BootID:e971d40e-99a9-4667-9f85-6e3c468425b7 Filesystems:[{Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8013e357-0fa2-4360-924b-18cbfe588b5c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:366 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e30bb07f1caa152986c84a26125c0e5346b0e8bc552b5156f435783366de1ce9/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:290 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:291 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90341f35811b436dab6f5eeb57758a8f6c6c901b850325b0566b18661bd1a6ed/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c112dbe2-b2e4-4762-8123-b9f161bda84a/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:348 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:282 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3889ae11c60344fc958acca1cee6adaec6562eb2ae83aa8b80d93ea893e7711b/userdata/shm DeviceMajor:0 DeviceMinor:367 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cfd93681f5f218e22c883a9e49151eb7f586674f01d276757174f9829a421c37/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/45ccae2d-1b91-4bbe-9598-94ad740116ec/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf/volumes/kubernetes.io~projected/kube-api-access-7z7pj DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45ccae2d-1b91-4bbe-9598-94ad740116ec/volumes/kubernetes.io~projected/kube-api-access-d5sb5 DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a341b7ef-207b-4443-b01e-9fcbc2ac0075/volumes/kubernetes.io~projected/kube-api-access-d895r DeviceMajor:0 DeviceMinor:326 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4cda2bf1ae154e1b02cf339cbcb9249ff3eb3aa791a8213ba323612fa10282cc/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:289 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/23de5b1c-4825-43ed-bff8-cdab8789989e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6385dafaa9f8c8e977f2d44f2491d84e9755c9c1ae80b00e04f2ca29bcdbf8b/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de8ddc3f-5d23-4cff-9f2e-9167700ff109/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:292 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c326058e8f2e9a4ae0387a68eab1d87ddbdf11405269f074395b5f275417f1c6/userdata/shm DeviceMajor:0 DeviceMinor:355 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c40aa314795318f3f4dd9529bcfa37edcc6c76c12978d657b607d6cd3ae69209/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/008928f8-b2ce-4f12-9681-21136d802ff6/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:236 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0a98773-9cb1-4637-9188-52094d47dceb/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5/volumes/kubernetes.io~projected/kube-api-access-q5sd9 DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/771d5ecbcb285260e210781715caeda168dd97a9820a0dfb0cb9a687f8a1c7ec/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3e9972f4-8918-41a0-8ae4-b2014d3897ca/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c979e36-8fff-4401-a991-1a661fd9c6e5/volumes/kubernetes.io~projected/kube-api-access-csj5l DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e2ac53be-96b4-4976-9581-b38cc97e1eb0/volumes/kubernetes.io~projected/kube-api-access-l6scd DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4174a924aa6c72d9f7eeab913165b762220275baf3d845126dc6629a256070b/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/008928f8-b2ce-4f12-9681-21136d802ff6/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:232 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/905a4f91743ddd1fb3ee8d85a3d61109230e811ed1b2c825e5e05dae8ccd8b7b/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4d92d32030cb453da5f69397058a394d5892d1066c08079fb94644e86839df7/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0de2915a-789c-4acc-84a6-c61987498b62/volumes/kubernetes.io~projected/kube-api-access-nxxqw DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e9972f4-8918-41a0-8ae4-b2014d3897ca/volumes/kubernetes.io~projected/kube-api-access-b4btm DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:277 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5b879a0b0afdc510709a56fedddbe3e63eee4eb9a2d869df77a7d9f3235cde6/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3e72d26b223d07b1cab9286bc95e64e8f5540868af1598407589ef50ee834aa/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed3d450f2f9f551f3fe9791bd70422e1170a01b7f13bf5a069d6f897d8939011/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/101b32f8-ea50-4c3d-8114-15ebe26eb6d0/volumes/kubernetes.io~projected/kube-api-access-d5z5s DeviceMajor:0 DeviceMinor:371 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/68ff550a-daae-421b-9dd6-e9b1f33585d1/volumes/kubernetes.io~projected/kube-api-access-9tcgh DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0d19f3a-000a-4911-96dd-cbf24e784627/volumes/kubernetes.io~projected/kube-api-access-w5rj9 DeviceMajor:0 DeviceMinor:294 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c112dbe2-b2e4-4762-8123-b9f161bda84a/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:353 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/e0a98773-9cb1-4637-9188-52094d47dceb/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c112dbe2-b2e4-4762-8123-b9f161bda84a/volumes/kubernetes.io~projected/kube-api-access-6k7xh DeviceMajor:0 DeviceMinor:354 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0a98773-9cb1-4637-9188-52094d47dceb/volumes/kubernetes.io~projected/kube-api-access-d6v4c DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c979e36-8fff-4401-a991-1a661fd9c6e5/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:361 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/53bd98ea-ea87-484a-b7b1-8c9070fc9b25/volumes/kubernetes.io~projected/kube-api-access-b9tds DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a341b7ef-207b-4443-b01e-9fcbc2ac0075/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:322 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c2836676-0428-4f2f-a801-0a25df87aee2/volumes/kubernetes.io~projected/kube-api-access-xbkb8 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57/volumes/kubernetes.io~projected/kube-api-access-k9fpq DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/823e2f643e9b2f48bd55729c3f8353509829c76f10b6e67696ff3b2964280b0f/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f3a89bbe93fdc66236bd2765050de4d26d191fef04c0713b85c04d3b08a37fc/userdata/shm DeviceMajor:0 DeviceMinor:327 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/008928f8-b2ce-4f12-9681-21136d802ff6/volumes/kubernetes.io~projected/kube-api-access-j8bxp DeviceMajor:0 DeviceMinor:237 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e80dc474dd09ed7516ce7158907a43843349e37465b9e0ab5ff913cfce5c9ce/userdata/shm DeviceMajor:0 DeviceMinor:302 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de8ddc3f-5d23-4cff-9f2e-9167700ff109/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1e3a66ef45c22aacb12406e3c79ac8b4fab71f1a5e5ceea73fa4e81516fdcf7/userdata/shm DeviceMajor:0 DeviceMinor:376 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d50ec904-b3c9-4677-9624-39cb202f65c9/volumes/kubernetes.io~projected/kube-api-access-pmjjp DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47ee8ad7afa297228f1b3adfeb887e1799deacaaad1a2cabc351691e1f0a39c5/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e59da0f23b582ab57c0b73f7043466ba70b4e7e2b61094e81723590699ea17f1/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0de2915a-789c-4acc-84a6-c61987498b62/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3e9972f4-8918-41a0-8ae4-b2014d3897ca/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de8ddc3f-5d23-4cff-9f2e-9167700ff109/volumes/kubernetes.io~projected/kube-api-access-f6hdz DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ff348abb90c20e755aed86db9e1bc9fb9c6fce67ea6ac04d107527b657466b5/userdata/shm DeviceMajor:0 DeviceMinor:369 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1f3a89bbe93fdc6 MacAddress:82:86:e3:43:be:8b Speed:10000 Mtu:8901} {Name:3889ae11c60344f MacAddress:7e:16:00:a9:05:a2 Speed:10000 Mtu:8901} {Name:3e80dc474dd09ed MacAddress:8a:1e:ba:77:85:fb Speed:10000 Mtu:8901} {Name:47ee8ad7afa2972 MacAddress:42:29:d7:8b:b3:af Speed:10000 Mtu:8901} {Name:4cda2bf1ae154e1 MacAddress:1a:63:18:15:cf:8e Speed:10000 Mtu:8901} {Name:4ff348abb90c20e MacAddress:12:f9:6d:56:3a:d4 Speed:10000 Mtu:8901} {Name:771d5ecbcb28526 MacAddress:a6:d3:38:00:39:85 Speed:10000 Mtu:8901} {Name:b1e3a66ef45c22a MacAddress:aa:ee:b3:d1:b0:f2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fd:5c:5d:b5 Speed:0 Mtu:9001} {Name:br-int MacAddress:7e:18:40:06:f9:81 Speed:0 Mtu:8901} {Name:c326058e8f2e9a4 MacAddress:f6:5b:b7:6f:29:b8 Speed:10000 Mtu:8901} {Name:e59da0f23b582ab MacAddress:26:7b:21:50:3c:35 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:fd:5c:5d:b5 Speed:0 Mtu:9001} {Name:f4d92d32030cb45 MacAddress:3e:12:3c:0a:9a:b9 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:66:9d:a6:a8:5f:0b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:da:94:fd:05:9b:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.477518    5408 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.477563    5408 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.477850    5408 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.477874    5408 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-19.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.478002    5408 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.478010    5408 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.478020    5408 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.478048    5408 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.478150    5408 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.478315    5408 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.478417    5408 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.478456    5408 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.478471    5408 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.478481    5408 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.478490    5408 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.478499    5408 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.480095    5408 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.480113    5408 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:30:45.481406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.480815    5408 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:30:45.483807 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.483790    5408 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:30:45.483807 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.483805    5408 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:30:45.483910 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.483898    5408 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 19:30:45.484112 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.484101    5408 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:30:45.484148 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.484130    5408 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:30:45.484221 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.484212    5408 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:30:45.484251 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.484228    5408 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:30:45.484251 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.484234    5408 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:30:45.484251 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.484239    5408 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:30:45.484251 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.484245    5408 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:30:45.484251 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.484251    5408 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:30:45.484387 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.484257    5408 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:30:45.484387 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.484262    5408 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:30:45.484387 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.484269    5408 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:30:45.484387 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.484275    5408 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:30:45.484387 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.484283    5408 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:30:45.484387 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.484293    5408 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:30:45.484387 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.484316    5408 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:30:45.484387 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.484322    5408 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:30:45.487578 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.487565    5408 server.go:1263] "Started kubelet"
Oct 30 19:30:45.487681 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.487636    5408 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:30:45.487745 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.487669    5408 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:30:45.487745 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.487740    5408 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:30:45.488331 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.488311    5408 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:30:45.488403 ip-10-0-141-19 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:30:45.489157 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.489139    5408 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:30:45.493049 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.491275    5408 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:30:45.493789 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.493773    5408 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:30:45.495321 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.494578    5408 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:24:35 +0000 UTC" deadline="2028-04-12 20:41:32.414351039 +0000 UTC"
Oct 30 19:30:45.495321 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.494625    5408 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:30:45.495321 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.494644    5408 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:30:45.495321 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.494792    5408 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:30:45.495321 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:30:45.494621    5408 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-19.us-east-2.compute.internal\" not found"
Oct 30 19:30:45.495321 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.494631    5408 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12721h10m46.919724646s"
Oct 30 19:30:45.499180 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.498829    5408 factory.go:55] Registering systemd factory
Oct 30 19:30:45.499180 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.498853    5408 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:30:45.499180 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499049    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45ccae2d-1b91-4bbe-9598-94ad740116ec" volumeName="kubernetes.io/secret/45ccae2d-1b91-4bbe-9598-94ad740116ec-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 19:30:45.499180 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499127    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de8ddc3f-5d23-4cff-9f2e-9167700ff109" volumeName="kubernetes.io/configmap/de8ddc3f-5d23-4cff-9f2e-9167700ff109-ovnkube-config" seLinuxMountContext=""
Oct 30 19:30:45.499180 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499143    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0a98773-9cb1-4637-9188-52094d47dceb" volumeName="kubernetes.io/projected/e0a98773-9cb1-4637-9188-52094d47dceb-kube-api-access-d6v4c" seLinuxMountContext=""
Oct 30 19:30:45.499180 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499158    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0d19f3a-000a-4911-96dd-cbf24e784627" volumeName="kubernetes.io/empty-dir/e0d19f3a-000a-4911-96dd-cbf24e784627-tmp" seLinuxMountContext=""
Oct 30 19:30:45.499180 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499176    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="008928f8-b2ce-4f12-9681-21136d802ff6" volumeName="kubernetes.io/configmap/008928f8-b2ce-4f12-9681-21136d802ff6-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:30:45.499562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499190    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf" volumeName="kubernetes.io/empty-dir/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-textfile" seLinuxMountContext=""
Oct 30 19:30:45.499562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499205    5408 factory.go:153] Registering CRI-O factory
Oct 30 19:30:45.499562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499207    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c112dbe2-b2e4-4762-8123-b9f161bda84a" volumeName="kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-config" seLinuxMountContext=""
Oct 30 19:30:45.499562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499219    5408 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:30:45.499562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499223    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d50ec904-b3c9-4677-9624-39cb202f65c9" volumeName="kubernetes.io/configmap/d50ec904-b3c9-4677-9624-39cb202f65c9-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 19:30:45.499562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499243    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0d19f3a-000a-4911-96dd-cbf24e784627" volumeName="kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 30 19:30:45.499562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499256    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="008928f8-b2ce-4f12-9681-21136d802ff6" volumeName="kubernetes.io/secret/008928f8-b2ce-4f12-9681-21136d802ff6-prometheus-operator-tls" seLinuxMountContext=""
Oct 30 19:30:45.499562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499268    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9972f4-8918-41a0-8ae4-b2014d3897ca" volumeName="kubernetes.io/projected/3e9972f4-8918-41a0-8ae4-b2014d3897ca-kube-api-access-b4btm" seLinuxMountContext=""
Oct 30 19:30:45.499562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499288    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53bd98ea-ea87-484a-b7b1-8c9070fc9b25" volumeName="kubernetes.io/projected/53bd98ea-ea87-484a-b7b1-8c9070fc9b25-kube-api-access-b9tds" seLinuxMountContext=""
Oct 30 19:30:45.499562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499295    5408 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:30:45.499562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499301    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf" volumeName="kubernetes.io/configmap/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 19:30:45.499562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499314    5408 factory.go:103] Registering Raw factory
Oct 30 19:30:45.499562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499482    5408 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499368    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9972f4-8918-41a0-8ae4-b2014d3897ca" volumeName="kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-service-ca" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499631    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c112dbe2-b2e4-4762-8123-b9f161bda84a" volumeName="kubernetes.io/projected/c112dbe2-b2e4-4762-8123-b9f161bda84a-kube-api-access-6k7xh" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499656    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2836676-0428-4f2f-a801-0a25df87aee2" volumeName="kubernetes.io/empty-dir/c2836676-0428-4f2f-a801-0a25df87aee2-tmp-dir" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499704    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf" volumeName="kubernetes.io/configmap/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499719    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53bd98ea-ea87-484a-b7b1-8c9070fc9b25" volumeName="kubernetes.io/configmap/53bd98ea-ea87-484a-b7b1-8c9070fc9b25-serviceca" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499737    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c979e36-8fff-4401-a991-1a661fd9c6e5" volumeName="kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499751    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57" volumeName="kubernetes.io/configmap/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499771    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4e933e1-4e6d-4a26-bb7d-68f366e02ac5" volumeName="kubernetes.io/projected/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-kube-api-access-q5sd9" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499805    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de8ddc3f-5d23-4cff-9f2e-9167700ff109" volumeName="kubernetes.io/secret/de8ddc3f-5d23-4cff-9f2e-9167700ff109-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499817    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0a98773-9cb1-4637-9188-52094d47dceb" volumeName="kubernetes.io/empty-dir/e0a98773-9cb1-4637-9188-52094d47dceb-etc-tuned" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499838    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0d19f3a-000a-4911-96dd-cbf24e784627" volumeName="kubernetes.io/configmap/e0d19f3a-000a-4911-96dd-cbf24e784627-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499851    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23de5b1c-4825-43ed-bff8-cdab8789989e" volumeName="kubernetes.io/configmap/23de5b1c-4825-43ed-bff8-cdab8789989e-konnectivity-ca" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499869    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="008928f8-b2ce-4f12-9681-21136d802ff6" volumeName="kubernetes.io/projected/008928f8-b2ce-4f12-9681-21136d802ff6-kube-api-access-j8bxp" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499882    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0de2915a-789c-4acc-84a6-c61987498b62" volumeName="kubernetes.io/projected/0de2915a-789c-4acc-84a6-c61987498b62-kube-api-access-nxxqw" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499894    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf" volumeName="kubernetes.io/secret/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-tls" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499932    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57" volumeName="kubernetes.io/projected/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-kube-api-access-k9fpq" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499950    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c112dbe2-b2e4-4762-8123-b9f161bda84a" volumeName="kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-oauth-serving-cert" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499965    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4e933e1-4e6d-4a26-bb7d-68f366e02ac5" volumeName="kubernetes.io/empty-dir/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-tmp-dir" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499980    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de8ddc3f-5d23-4cff-9f2e-9167700ff109" volumeName="kubernetes.io/projected/de8ddc3f-5d23-4cff-9f2e-9167700ff109-kube-api-access-f6hdz" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.499998    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45ccae2d-1b91-4bbe-9598-94ad740116ec" volumeName="kubernetes.io/configmap/45ccae2d-1b91-4bbe-9598-94ad740116ec-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500012    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a341b7ef-207b-4443-b01e-9fcbc2ac0075" volumeName="kubernetes.io/projected/a341b7ef-207b-4443-b01e-9fcbc2ac0075-kube-api-access-d895r" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500048    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23de5b1c-4825-43ed-bff8-cdab8789989e" volumeName="kubernetes.io/secret/23de5b1c-4825-43ed-bff8-cdab8789989e-agent-certs" seLinuxMountContext=""
Oct 30 19:30:45.500158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500011    5408 manager.go:319] Starting recovery of all containers
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500342    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c112dbe2-b2e4-4762-8123-b9f161bda84a" volumeName="kubernetes.io/empty-dir/c112dbe2-b2e4-4762-8123-b9f161bda84a-tmp" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500372    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4e933e1-4e6d-4a26-bb7d-68f366e02ac5" volumeName="kubernetes.io/configmap/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-config-volume" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500527    5408 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500383    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0d19f3a-000a-4911-96dd-cbf24e784627" volumeName="kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500559    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="101b32f8-ea50-4c3d-8114-15ebe26eb6d0" volumeName="kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500587    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9972f4-8918-41a0-8ae4-b2014d3897ca" volumeName="kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-oauth-config" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500604    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a341b7ef-207b-4443-b01e-9fcbc2ac0075" volumeName="kubernetes.io/secret/a341b7ef-207b-4443-b01e-9fcbc2ac0075-monitoring-plugin-cert" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500612    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d50ec904-b3c9-4677-9624-39cb202f65c9" volumeName="kubernetes.io/projected/d50ec904-b3c9-4677-9624-39cb202f65c9-kube-api-access-pmjjp" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500626    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0d19f3a-000a-4911-96dd-cbf24e784627" volumeName="kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500635    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0d19f3a-000a-4911-96dd-cbf24e784627" volumeName="kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-grpc-tls" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500644    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2ac53be-96b4-4976-9581-b38cc97e1eb0" volumeName="kubernetes.io/configmap/e2ac53be-96b4-4976-9581-b38cc97e1eb0-iptables-alerter-script" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500655    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf" volumeName="kubernetes.io/secret/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500663    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9972f4-8918-41a0-8ae4-b2014d3897ca" volumeName="kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-config" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500675    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9972f4-8918-41a0-8ae4-b2014d3897ca" volumeName="kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-serving-cert" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500684    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d50ec904-b3c9-4677-9624-39cb202f65c9" volumeName="kubernetes.io/configmap/d50ec904-b3c9-4677-9624-39cb202f65c9-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500693    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0de2915a-789c-4acc-84a6-c61987498b62" volumeName="kubernetes.io/secret/0de2915a-789c-4acc-84a6-c61987498b62-cert" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500704    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0d19f3a-000a-4911-96dd-cbf24e784627" volumeName="kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500714    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45ccae2d-1b91-4bbe-9598-94ad740116ec" volumeName="kubernetes.io/empty-dir/45ccae2d-1b91-4bbe-9598-94ad740116ec-data-volume" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500727    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9972f4-8918-41a0-8ae4-b2014d3897ca" volumeName="kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-oauth-serving-cert" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500738    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9972f4-8918-41a0-8ae4-b2014d3897ca" volumeName="kubernetes.io/empty-dir/3e9972f4-8918-41a0-8ae4-b2014d3897ca-tmp" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500762    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57" volumeName="kubernetes.io/configmap/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-daemon-config" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500775    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c112dbe2-b2e4-4762-8123-b9f161bda84a" volumeName="kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-service-ca" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500799    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de8ddc3f-5d23-4cff-9f2e-9167700ff109" volumeName="kubernetes.io/configmap/de8ddc3f-5d23-4cff-9f2e-9167700ff109-env-overrides" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500808    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de8ddc3f-5d23-4cff-9f2e-9167700ff109" volumeName="kubernetes.io/configmap/de8ddc3f-5d23-4cff-9f2e-9167700ff109-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500819    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0a98773-9cb1-4637-9188-52094d47dceb" volumeName="kubernetes.io/empty-dir/e0a98773-9cb1-4637-9188-52094d47dceb-tmp" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500828    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0d19f3a-000a-4911-96dd-cbf24e784627" volumeName="kubernetes.io/projected/e0d19f3a-000a-4911-96dd-cbf24e784627-kube-api-access-w5rj9" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500840    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="008928f8-b2ce-4f12-9681-21136d802ff6" volumeName="kubernetes.io/secret/008928f8-b2ce-4f12-9681-21136d802ff6-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500848    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d50ec904-b3c9-4677-9624-39cb202f65c9" volumeName="kubernetes.io/configmap/d50ec904-b3c9-4677-9624-39cb202f65c9-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500859    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2ac53be-96b4-4976-9581-b38cc97e1eb0" volumeName="kubernetes.io/projected/e2ac53be-96b4-4976-9581-b38cc97e1eb0-kube-api-access-l6scd" seLinuxMountContext=""
Oct 30 19:30:45.500850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500868    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf" volumeName="kubernetes.io/projected/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-kube-api-access-7z7pj" seLinuxMountContext=""
Oct 30 19:30:45.502112 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500877    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45ccae2d-1b91-4bbe-9598-94ad740116ec" volumeName="kubernetes.io/projected/45ccae2d-1b91-4bbe-9598-94ad740116ec-kube-api-access-d5sb5" seLinuxMountContext=""
Oct 30 19:30:45.502112 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500906    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8013e357-0fa2-4360-924b-18cbfe588b5c" volumeName="kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret" seLinuxMountContext=""
Oct 30 19:30:45.502112 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500921    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c112dbe2-b2e4-4762-8123-b9f161bda84a" volumeName="kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:30:45.502112 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500930    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c112dbe2-b2e4-4762-8123-b9f161bda84a" volumeName="kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-oauth-config" seLinuxMountContext=""
Oct 30 19:30:45.502112 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500941    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2836676-0428-4f2f-a801-0a25df87aee2" volumeName="kubernetes.io/projected/c2836676-0428-4f2f-a801-0a25df87aee2-kube-api-access-xbkb8" seLinuxMountContext=""
Oct 30 19:30:45.502112 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500949    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0d19f3a-000a-4911-96dd-cbf24e784627" volumeName="kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 19:30:45.502112 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500959    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68ff550a-daae-421b-9dd6-e9b1f33585d1" volumeName="kubernetes.io/projected/68ff550a-daae-421b-9dd6-e9b1f33585d1-kube-api-access-9tcgh" seLinuxMountContext=""
Oct 30 19:30:45.502112 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500969    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c979e36-8fff-4401-a991-1a661fd9c6e5" volumeName="kubernetes.io/projected/7c979e36-8fff-4401-a991-1a661fd9c6e5-kube-api-access-csj5l" seLinuxMountContext=""
Oct 30 19:30:45.502112 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500978    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c112dbe2-b2e4-4762-8123-b9f161bda84a" volumeName="kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-serving-cert" seLinuxMountContext=""
Oct 30 19:30:45.502112 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.500992    5408 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4e933e1-4e6d-4a26-bb7d-68f366e02ac5" volumeName="kubernetes.io/secret/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-metrics-tls" seLinuxMountContext=""
Oct 30 19:30:45.502112 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.501224    5408 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:30:45.502112 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.501233    5408 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:30:45.502665 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.502636    5408 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:30:45.503172 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.503150    5408 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:30:45.506114 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.506096    5408 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 19:30:45.522134 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.522076    5408 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:30:45.523595 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.523561    5408 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:30:45.523595 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.523592    5408 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:30:45.523763 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.523612    5408 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:30:45.523763 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:30:45.523660    5408 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 19:30:45.528287 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.528263    5408 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:30:45.564732 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.564696    5408 generic.go:356] "Generic (PLEG): container finished" podID="d4d61e23ca703103c56d5c7a000f30c8" containerID="32bdffa2472c4e691d97a404f2c6ef5b915b01d0306683e281ad26c4e97ae4da" exitCode=0
Oct 30 19:30:45.577440 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.577384    5408 generic.go:356] "Generic (PLEG): container finished" podID="d50ec904-b3c9-4677-9624-39cb202f65c9" containerID="b820bd4d290ed7f6fc9913def85165541c71f2f0f5f42b907496d57b336b771d" exitCode=0
Oct 30 19:30:45.577594 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.577444    5408 generic.go:356] "Generic (PLEG): container finished" podID="d50ec904-b3c9-4677-9624-39cb202f65c9" containerID="ef7734784ff9c4000cd6d12427135f50dd1adf0a2d975ccd2021bac7da0768a4" exitCode=0
Oct 30 19:30:45.577594 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.577467    5408 generic.go:356] "Generic (PLEG): container finished" podID="d50ec904-b3c9-4677-9624-39cb202f65c9" containerID="c3bfefde28bf8ece52936d9c16f70cb124846f9601604853ea603504807ada05" exitCode=0
Oct 30 19:30:45.577594 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.577483    5408 generic.go:356] "Generic (PLEG): container finished" podID="d50ec904-b3c9-4677-9624-39cb202f65c9" containerID="df72d35bf995a8dca2dd05eb5df90c423caafc8099b0fc54bd34cc83a499bbdb" exitCode=0
Oct 30 19:30:45.577594 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.577503    5408 generic.go:356] "Generic (PLEG): container finished" podID="d50ec904-b3c9-4677-9624-39cb202f65c9" containerID="326d0682233a45834c22a6f31b1f7c222c06a13c580555894723e16e649c9218" exitCode=0
Oct 30 19:30:45.577594 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.577524    5408 generic.go:356] "Generic (PLEG): container finished" podID="d50ec904-b3c9-4677-9624-39cb202f65c9" containerID="8c64d40d5554ebffab25ca34a05655a9ee4aaa9547400bd1335e98cfac502ff9" exitCode=0
Oct 30 19:30:45.584868 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.584834    5408 generic.go:356] "Generic (PLEG): container finished" podID="0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf" containerID="026a7c68a8b24384c1d684c5e8bf908d0746343d8f6ad743e5440db8dc719b2c" exitCode=0
Oct 30 19:30:45.598796 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.598768    5408 manager.go:324] Recovery completed
Oct 30 19:30:45.621212 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.621178    5408 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:30:45.621212 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.621199    5408 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:30:45.621423 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.621221    5408 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:30:45.621423 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.621406    5408 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 19:30:45.621541 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.621422    5408 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 19:30:45.621541 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.621446    5408 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 19:30:45.621541 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.621456    5408 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 19:30:45.621541 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.621464    5408 policy_none.go:50] "Start"
Oct 30 19:30:45.621541 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.621476    5408 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:30:45.621541 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.621489    5408 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:30:45.621800 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.621664    5408 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 19:30:45.621800 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.621675    5408 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 19:30:45.621800 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.621683    5408 policy_none.go:44] "Start"
Oct 30 19:30:45.623759 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:30:45.623705    5408 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 19:30:45.626179 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.626151    5408 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:30:45.629591 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.629577    5408 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:30:45.630069 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:30:45.629604    5408 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:30:45.630156 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.630072    5408 server.go:87] "Starting device plugin registration server"
Oct 30 19:30:45.630367 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.630354    5408 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:30:45.630440 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.630368    5408 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:30:45.630488 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.630455    5408 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:30:45.630603 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.630568    5408 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:30:45.630658 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.630616    5408 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:30:45.630658 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.630624    5408 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:30:45.630795 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.630675    5408 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:30:45.630600612Z","UUID":"51a9d421-1394-459f-ad8d-26bd6f607345","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:30:45.632569 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:30:45.632540    5408 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:30:45.632747 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.632730    5408 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:30:45.632833 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.632755    5408 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:30:45.731198 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.731161    5408 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:30:45.733506 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.733489    5408 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:30:45.733635 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.733519    5408 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:30:45.733635 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.733530    5408 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-19.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:30:45.733635 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.733569    5408 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:45.746363 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.746331    5408 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:45.746524 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.746437    5408 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:45.824820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.824755    5408 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal"]
Oct 30 19:30:45.825407 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.825359    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal" event={"ID":"d4d61e23ca703103c56d5c7a000f30c8","Type":"ContainerStarted","Data":"a0cab43135b6401705ca48d899f27ff1dd14bbded046cc530250836218fbd16e"}
Oct 30 19:30:45.825496 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.825413    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal" event={"ID":"d4d61e23ca703103c56d5c7a000f30c8","Type":"ContainerDied","Data":"32bdffa2472c4e691d97a404f2c6ef5b915b01d0306683e281ad26c4e97ae4da"}
Oct 30 19:30:45.825496 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.825428    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal" event={"ID":"d4d61e23ca703103c56d5c7a000f30c8","Type":"ContainerStarted","Data":"c3e72d26b223d07b1cab9286bc95e64e8f5540868af1598407589ef50ee834aa"}
Oct 30 19:30:45.825496 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.825440    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal" event={"ID":"7dbf81fa0f84f4c1de7386fb2838c009","Type":"ContainerStarted","Data":"3c1e0992e64794dda6a03b52961bbbeea8b8c57dba17fc3bee4785e3d301d968"}
Oct 30 19:30:45.825496 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.825448    5408 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:45.825621 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.825534    5408 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:45.825621 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.825452    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal" event={"ID":"7dbf81fa0f84f4c1de7386fb2838c009","Type":"ContainerStarted","Data":"f5b879a0b0afdc510709a56fedddbe3e63eee4eb9a2d869df77a7d9f3235cde6"}
Oct 30 19:30:45.836502 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.836488    5408 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:30:45.836558 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:30:45.836527    5408 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:45.837117 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.837104    5408 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:30:45.837150 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:30:45.837138    5408 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:45.908235 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.908168    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4d61e23ca703103c56d5c7a000f30c8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal\" (UID: \"d4d61e23ca703103c56d5c7a000f30c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:45.908235 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.908199    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4d61e23ca703103c56d5c7a000f30c8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal\" (UID: \"d4d61e23ca703103c56d5c7a000f30c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:45.908235 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:45.908224    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dbf81fa0f84f4c1de7386fb2838c009-config\") pod \"kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal\" (UID: \"7dbf81fa0f84f4c1de7386fb2838c009\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:46.008762 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.008725    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dbf81fa0f84f4c1de7386fb2838c009-config\") pod \"kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal\" (UID: \"7dbf81fa0f84f4c1de7386fb2838c009\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:46.008762 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.008765    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4d61e23ca703103c56d5c7a000f30c8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal\" (UID: \"d4d61e23ca703103c56d5c7a000f30c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:46.008936 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.008787    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4d61e23ca703103c56d5c7a000f30c8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal\" (UID: \"d4d61e23ca703103c56d5c7a000f30c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:46.008936 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.008829    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dbf81fa0f84f4c1de7386fb2838c009-config\") pod \"kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal\" (UID: \"7dbf81fa0f84f4c1de7386fb2838c009\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:46.008936 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.008834    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4d61e23ca703103c56d5c7a000f30c8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal\" (UID: \"d4d61e23ca703103c56d5c7a000f30c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:46.008936 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.008834    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4d61e23ca703103c56d5c7a000f30c8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal\" (UID: \"d4d61e23ca703103c56d5c7a000f30c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:46.478679 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.478644    5408 apiserver.go:52] "Watching apiserver"
Oct 30 19:30:46.490002 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.489969    5408 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:30:46.490559 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.490534    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-854b9d69db-nvllp","openshift-dns/node-resolver-gmrbv","openshift-monitoring/node-exporter-rb876","openshift-ingress-canary/ingress-canary-2c2l4","openshift-multus/multus-additional-cni-plugins-qmqjk","openshift-multus/network-metrics-daemon-mwcmw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-vwkv5","openshift-multus/multus-d6gqm","openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7","openshift-console/console-698967b46-ghthb","kube-system/konnectivity-agent-kspfd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn","openshift-dns/dns-default-dkkwh","kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-bkdq5","openshift-image-registry/node-ca-h5m52","openshift-monitoring/thanos-querier-67c75b6564-g8dq2","openshift-insights/insights-runtime-extractor-rkc6x","openshift-monitoring/prometheus-operator-9cd89b74c-9s95l","openshift-network-operator/iptables-alerter-5n649","openshift-ovn-kubernetes/ovnkube-node-gjswr","kube-system/global-pull-secret-syncer-d2hv2"]
Oct 30 19:30:46.497125 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.497091    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2prmp\""
Oct 30 19:30:46.497125 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.497105    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xs4jl\""
Oct 30 19:30:46.497320 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.497248    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z724k\""
Oct 30 19:30:46.497320 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.497255    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q2kw5\""
Oct 30 19:30:46.497490 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.497468    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zqbbj\""
Oct 30 19:30:46.497563 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.497547    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wp4kx\""
Oct 30 19:30:46.499219 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499173    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:30:46.499480 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499411    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:30:46.499671 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499462    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:30:46.499671 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499666    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:30:46.499820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499519    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:30:46.499820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499474    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:30:46.499820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499540    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:30:46.499820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499727    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:30:46.499820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499566    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:30:46.499820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499748    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:30:46.499820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499609    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:30:46.499820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499612    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:30:46.499820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499669    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:30:46.499820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499525    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:30:46.499820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499730    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:30:46.499820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499575    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:30:46.499820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499743    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:30:46.499820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499582    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:30:46.499820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.499802    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:30:46.500632 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.500613    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:30:46.500714 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.500637    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:30:46.500756 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.500727    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:30:46.500882 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.500866    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:30:46.500959 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.500883    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:30:46.500959 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.500916    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:30:46.500959 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.500925    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:30:46.500959 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.500917    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:30:46.501386 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.501365    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:30:46.501483 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.501406    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:30:46.501515 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.501375    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p6d49\""
Oct 30 19:30:46.501545 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.501366    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:30:46.502330 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.502310    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fbg5p\""
Oct 30 19:30:46.502468 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.502450    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ttn9c\""
Oct 30 19:30:46.502658 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.502637    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zsvdk\""
Oct 30 19:30:46.502799 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.502779    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-svfmj\""
Oct 30 19:30:46.502862 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.502842    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:30:46.502950 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.502936    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:30:46.503005 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.502968    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pn76m\""
Oct 30 19:30:46.503072 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.503062    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:30:46.503185 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.503149    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2cs46\""
Oct 30 19:30:46.503185 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.503164    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:30:46.503347 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.503292    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-q9kwd\""
Oct 30 19:30:46.503347 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.503318    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8hwq2\""
Oct 30 19:30:46.510872 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.510849    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5z5s\" (UniqueName: \"kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s\") pod \"network-check-target-vwkv5\" (UID: \"101b32f8-ea50-4c3d-8114-15ebe26eb6d0\") " pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:30:46.510958 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.510884    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:46.510958 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.510912    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-hostroot\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.510958 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.510936    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.511092 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.510960    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-device-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.511092 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.510984    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tcgh\" (UniqueName: \"kubernetes.io/projected/68ff550a-daae-421b-9dd6-e9b1f33585d1-kube-api-access-9tcgh\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.511092 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511009    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-socket-dir-parent\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.511092 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511065    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-run-k8s-cni-cncf-io\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.511270 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511189    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-daemon-config\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.511270 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511225    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9fpq\" (UniqueName: \"kubernetes.io/projected/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-kube-api-access-k9fpq\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.511270 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511247    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8013e357-0fa2-4360-924b-18cbfe588b5c-original-pull-secret\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:46.511270 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511255    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-kubernetes\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.511449 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511284    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-node-log\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.511449 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511310    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de8ddc3f-5d23-4cff-9f2e-9167700ff109-env-overrides\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.511449 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511361    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de8ddc3f-5d23-4cff-9f2e-9167700ff109-ovn-node-metrics-cert\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.511449 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511377    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-daemon-config\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.511449 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511417    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2836676-0428-4f2f-a801-0a25df87aee2-tmp-dir\") pod \"node-resolver-gmrbv\" (UID: \"c2836676-0428-4f2f-a801-0a25df87aee2\") " pod="openshift-dns/node-resolver-gmrbv"
Oct 30 19:30:46.511665 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511468    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-sysconfig\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.511665 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511479    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2836676-0428-4f2f-a801-0a25df87aee2-tmp-dir\") pod \"node-resolver-gmrbv\" (UID: \"c2836676-0428-4f2f-a801-0a25df87aee2\") " pod="openshift-dns/node-resolver-gmrbv"
Oct 30 19:30:46.511665 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511501    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de8ddc3f-5d23-4cff-9f2e-9167700ff109-env-overrides\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.511665 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511515    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-host\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.511665 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511557    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e2ac53be-96b4-4976-9581-b38cc97e1eb0-iptables-alerter-script\") pod \"iptables-alerter-5n649\" (UID: \"e2ac53be-96b4-4976-9581-b38cc97e1eb0\") " pod="openshift-network-operator/iptables-alerter-5n649"
Oct 30 19:30:46.511665 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511598    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2ac53be-96b4-4976-9581-b38cc97e1eb0-host-slash\") pod \"iptables-alerter-5n649\" (UID: \"e2ac53be-96b4-4976-9581-b38cc97e1eb0\") " pod="openshift-network-operator/iptables-alerter-5n649"
Oct 30 19:30:46.511665 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511650    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de8ddc3f-5d23-4cff-9f2e-9167700ff109-ovn-node-metrics-cert\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.511665 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511656    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/23de5b1c-4825-43ed-bff8-cdab8789989e-agent-certs\") pod \"konnectivity-agent-kspfd\" (UID: \"23de5b1c-4825-43ed-bff8-cdab8789989e\") " pod="kube-system/konnectivity-agent-kspfd"
Oct 30 19:30:46.512000 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511676    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d50ec904-b3c9-4677-9624-39cb202f65c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:30:46.512000 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511761    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e2ac53be-96b4-4976-9581-b38cc97e1eb0-iptables-alerter-script\") pod \"iptables-alerter-5n649\" (UID: \"e2ac53be-96b4-4976-9581-b38cc97e1eb0\") " pod="openshift-network-operator/iptables-alerter-5n649"
Oct 30 19:30:46.512000 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511782    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d50ec904-b3c9-4677-9624-39cb202f65c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:30:46.512000 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511797    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d50ec904-b3c9-4677-9624-39cb202f65c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:30:46.512000 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511814    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-cni-dir\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.512000 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511837    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-modprobe-d\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.512000 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511880    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-sys\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.512000 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511907    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-run-ovn\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.512000 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511918    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/23de5b1c-4825-43ed-bff8-cdab8789989e-agent-certs\") pod \"konnectivity-agent-kspfd\" (UID: \"23de5b1c-4825-43ed-bff8-cdab8789989e\") " pod="kube-system/konnectivity-agent-kspfd"
Oct 30 19:30:46.512000 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511929    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-log-socket\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.512000 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.511983    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6hdz\" (UniqueName: \"kubernetes.io/projected/de8ddc3f-5d23-4cff-9f2e-9167700ff109-kube-api-access-f6hdz\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.512442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512009    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9tds\" (UniqueName: \"kubernetes.io/projected/53bd98ea-ea87-484a-b7b1-8c9070fc9b25-kube-api-access-b9tds\") pod \"node-ca-h5m52\" (UID: \"53bd98ea-ea87-484a-b7b1-8c9070fc9b25\") " pod="openshift-image-registry/node-ca-h5m52"
Oct 30 19:30:46.512442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512051    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.512442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512079    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de8ddc3f-5d23-4cff-9f2e-9167700ff109-ovnkube-config\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.512442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512106    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-var-lib-kubelet\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.512442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512140    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-conf-dir\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.512442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512168    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-run\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.512442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512197    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-var-lib-kubelet\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.512442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512227    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbkb8\" (UniqueName: \"kubernetes.io/projected/c2836676-0428-4f2f-a801-0a25df87aee2-kube-api-access-xbkb8\") pod \"node-resolver-gmrbv\" (UID: \"c2836676-0428-4f2f-a801-0a25df87aee2\") " pod="openshift-dns/node-resolver-gmrbv"
Oct 30 19:30:46.512442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512243    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-var-lib-cni-bin\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.512442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512257    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-var-lib-cni-multus\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.512442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512277    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-run-multus-certs\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.512442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512312    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de8ddc3f-5d23-4cff-9f2e-9167700ff109-ovnkube-config\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.512442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512358    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-cni-bin\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.512442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512399    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-slash\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.512442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512444    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-run-netns\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.512970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512469    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-run-openvswitch\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.512970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512515    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c2836676-0428-4f2f-a801-0a25df87aee2-hosts-file\") pod \"node-resolver-gmrbv\" (UID: \"c2836676-0428-4f2f-a801-0a25df87aee2\") " pod="openshift-dns/node-resolver-gmrbv"
Oct 30 19:30:46.512970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512558    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-etc-openvswitch\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.512970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512585    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-run-ovn-kubernetes\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.512970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512604    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de8ddc3f-5d23-4cff-9f2e-9167700ff109-ovnkube-script-lib\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.512970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512642    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8013e357-0fa2-4360-924b-18cbfe588b5c-kubelet-config\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:46.512970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512682    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d50ec904-b3c9-4677-9624-39cb202f65c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:30:46.512970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512709    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-socket-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.512970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512752    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de8ddc3f-5d23-4cff-9f2e-9167700ff109-ovnkube-script-lib\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.512970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512820    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e0a98773-9cb1-4637-9188-52094d47dceb-etc-tuned\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.512970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512858    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0a98773-9cb1-4637-9188-52094d47dceb-tmp\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.512970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512871    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d50ec904-b3c9-4677-9624-39cb202f65c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:30:46.512970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512883    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e0a98773-9cb1-4637-9188-52094d47dceb-etc-tuned\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.512970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512903    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6v4c\" (UniqueName: \"kubernetes.io/projected/e0a98773-9cb1-4637-9188-52094d47dceb-kube-api-access-d6v4c\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.512970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512926    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0a98773-9cb1-4637-9188-52094d47dceb-tmp\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.512970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512950    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-var-lib-openvswitch\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.512970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.512977    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d50ec904-b3c9-4677-9624-39cb202f65c9-system-cni-dir\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:30:46.513477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513002    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d50ec904-b3c9-4677-9624-39cb202f65c9-cnibin\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:30:46.513477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513050    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjjp\" (UniqueName: \"kubernetes.io/projected/d50ec904-b3c9-4677-9624-39cb202f65c9-kube-api-access-pmjjp\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:30:46.513477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513124    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.513477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513157    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53bd98ea-ea87-484a-b7b1-8c9070fc9b25-host\") pod \"node-ca-h5m52\" (UID: \"53bd98ea-ea87-484a-b7b1-8c9070fc9b25\") " pod="openshift-image-registry/node-ca-h5m52"
Oct 30 19:30:46.513477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513201    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-system-cni-dir\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.513477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513231    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-run-netns\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.513477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513271    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-lib-modules\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.513477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513303    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-etc-kubernetes\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.513477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513334    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-sys-fs\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.513477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513356    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs\") pod \"network-metrics-daemon-mwcmw\" (UID: \"7c979e36-8fff-4401-a991-1a661fd9c6e5\") " pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:30:46.513477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513374    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csj5l\" (UniqueName: \"kubernetes.io/projected/7c979e36-8fff-4401-a991-1a661fd9c6e5-kube-api-access-csj5l\") pod \"network-metrics-daemon-mwcmw\" (UID: \"7c979e36-8fff-4401-a991-1a661fd9c6e5\") " pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:30:46.513477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513420    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-sysctl-conf\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.513477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513443    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-cnibin\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.513477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513460    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53bd98ea-ea87-484a-b7b1-8c9070fc9b25-serviceca\") pod \"node-ca-h5m52\" (UID: \"53bd98ea-ea87-484a-b7b1-8c9070fc9b25\") " pod="openshift-image-registry/node-ca-h5m52"
Oct 30 19:30:46.513931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513498    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8013e357-0fa2-4360-924b-18cbfe588b5c-dbus\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:46.513931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513586    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c979e36-8fff-4401-a991-1a661fd9c6e5-metrics-certs\") pod \"network-metrics-daemon-mwcmw\" (UID: \"7c979e36-8fff-4401-a991-1a661fd9c6e5\") " pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:30:46.513931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513598    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d50ec904-b3c9-4677-9624-39cb202f65c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:30:46.513931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513624    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-cni-binary-copy\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.513931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513641    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-sysctl-d\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.513931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513656    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-systemd\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.513931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513675    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-kubelet\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.513931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513730    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-run-systemd\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.513931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513802    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/23de5b1c-4825-43ed-bff8-cdab8789989e-konnectivity-ca\") pod \"konnectivity-agent-kspfd\" (UID: \"23de5b1c-4825-43ed-bff8-cdab8789989e\") " pod="kube-system/konnectivity-agent-kspfd"
Oct 30 19:30:46.513931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513825    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-registration-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.513931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513843    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-systemd-units\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.513931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513866    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-cni-binary-copy\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.513931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513873    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-cni-netd\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.513931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513908    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6scd\" (UniqueName: \"kubernetes.io/projected/e2ac53be-96b4-4976-9581-b38cc97e1eb0-kube-api-access-l6scd\") pod \"iptables-alerter-5n649\" (UID: \"e2ac53be-96b4-4976-9581-b38cc97e1eb0\") " pod="openshift-network-operator/iptables-alerter-5n649"
Oct 30 19:30:46.514388 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513992    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d50ec904-b3c9-4677-9624-39cb202f65c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:30:46.514388 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.513994    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/23de5b1c-4825-43ed-bff8-cdab8789989e-konnectivity-ca\") pod \"konnectivity-agent-kspfd\" (UID: \"23de5b1c-4825-43ed-bff8-cdab8789989e\") " pod="kube-system/konnectivity-agent-kspfd"
Oct 30 19:30:46.514388 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.514151    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/53bd98ea-ea87-484a-b7b1-8c9070fc9b25-serviceca\") pod \"node-ca-h5m52\" (UID: \"53bd98ea-ea87-484a-b7b1-8c9070fc9b25\") " pod="openshift-image-registry/node-ca-h5m52"
Oct 30 19:30:46.531435 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.531401    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tcgh\" (UniqueName: \"kubernetes.io/projected/68ff550a-daae-421b-9dd6-e9b1f33585d1-kube-api-access-9tcgh\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.531435 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.531415    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5z5s\" (UniqueName: \"kubernetes.io/projected/101b32f8-ea50-4c3d-8114-15ebe26eb6d0-kube-api-access-d5z5s\") pod \"network-check-target-vwkv5\" (UID: \"101b32f8-ea50-4c3d-8114-15ebe26eb6d0\") " pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:30:46.531570 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.531382    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6scd\" (UniqueName: \"kubernetes.io/projected/e2ac53be-96b4-4976-9581-b38cc97e1eb0-kube-api-access-l6scd\") pod \"iptables-alerter-5n649\" (UID: \"e2ac53be-96b4-4976-9581-b38cc97e1eb0\") " pod="openshift-network-operator/iptables-alerter-5n649"
Oct 30 19:30:46.532016 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.531983    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csj5l\" (UniqueName: \"kubernetes.io/projected/7c979e36-8fff-4401-a991-1a661fd9c6e5-kube-api-access-csj5l\") pod \"network-metrics-daemon-mwcmw\" (UID: \"7c979e36-8fff-4401-a991-1a661fd9c6e5\") " pod="openshift-multus/network-metrics-daemon-mwcmw"
Oct 30 19:30:46.532104 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.532014    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9tds\" (UniqueName: \"kubernetes.io/projected/53bd98ea-ea87-484a-b7b1-8c9070fc9b25-kube-api-access-b9tds\") pod \"node-ca-h5m52\" (UID: \"53bd98ea-ea87-484a-b7b1-8c9070fc9b25\") " pod="openshift-image-registry/node-ca-h5m52"
Oct 30 19:30:46.532104 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.532080    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9fpq\" (UniqueName: \"kubernetes.io/projected/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-kube-api-access-k9fpq\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.532391 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.532374    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6hdz\" (UniqueName: \"kubernetes.io/projected/de8ddc3f-5d23-4cff-9f2e-9167700ff109-kube-api-access-f6hdz\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.532511 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.532493    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbkb8\" (UniqueName: \"kubernetes.io/projected/c2836676-0428-4f2f-a801-0a25df87aee2-kube-api-access-xbkb8\") pod \"node-resolver-gmrbv\" (UID: \"c2836676-0428-4f2f-a801-0a25df87aee2\") " pod="openshift-dns/node-resolver-gmrbv"
Oct 30 19:30:46.532542 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.532508    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjjp\" (UniqueName: \"kubernetes.io/projected/d50ec904-b3c9-4677-9624-39cb202f65c9-kube-api-access-pmjjp\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:30:46.532855 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.532836    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6v4c\" (UniqueName: \"kubernetes.io/projected/e0a98773-9cb1-4637-9188-52094d47dceb-kube-api-access-d6v4c\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.588899 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.588867    5408 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:46.589102 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.588993    5408 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:46.595801 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.595744    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 19:30:46.595801 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.595804    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:30:46.596183 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.595879    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:30:46.596183 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.595876    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 19:30:46.596183 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.595906    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:30:46.596183 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.595919    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:30:46.596183 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.595930    5408 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:30:46.596183 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.595979    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 19:30:46.596183 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.595936    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:30:46.596183 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.596041    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 19:30:46.596183 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.596048    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:30:46.596183 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.595935    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:30:46.596183 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.596160    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:30:46.596631 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.596249    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 19:30:46.596631 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.596379    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 19:30:46.597639 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.597621    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 19:30:46.598260 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.598233    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 19:30:46.598325 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.598310    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 19:30:46.599514 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.599429    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 19:30:46.599514 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.599442    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ch4qh\""
Oct 30 19:30:46.599514 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.599494    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tr8z5\""
Oct 30 19:30:46.599514 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.599508    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 19:30:46.599514 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.599535    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 19:30:46.599879 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.599577    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 19:30:46.599879 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.599640    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 19:30:46.599879 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.599720    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 19:30:46.599879 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.599727    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 19:30:46.599879 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.599722    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bl6kg3usqeas\""
Oct 30 19:30:46.599879 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.599756    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 19:30:46.599879 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.599770    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 19:30:46.599879 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.599777    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nr7cs\""
Oct 30 19:30:46.599879 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.599770    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-f48l9\""
Oct 30 19:30:46.599879 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.599796    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6jgjw\""
Oct 30 19:30:46.599879 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.599828    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 19:30:46.599879 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.599870    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 19:30:46.601375 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.601362    5408 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:30:46.601425 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:30:46.601404    5408 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:46.601814 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.601799    5408 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:30:46.601882 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:30:46.601838    5408 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-19.us-east-2.compute.internal"
Oct 30 19:30:46.614426 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614406    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.614489 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614438    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-wtmp\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.614489 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614458    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-var-lib-cni-bin\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.614489 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614484    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-run-multus-certs\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.614596 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614483    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.614596 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614526    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-cni-bin\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.614596 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614549    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-var-lib-cni-bin\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.614596 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614559    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-run-multus-certs\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.614596 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614551    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8013e357-0fa2-4360-924b-18cbfe588b5c-kubelet-config\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:46.614596 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614577    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8013e357-0fa2-4360-924b-18cbfe588b5c-kubelet-config\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:46.614596 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614592    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-config-volume\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:46.614880 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614611    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-metrics-tls\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:46.614880 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614631    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45ccae2d-1b91-4bbe-9598-94ad740116ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:46.614880 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614649    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:46.614880 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614628    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-cni-bin\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.614880 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614670    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-service-ca\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:46.614880 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614783    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c2836676-0428-4f2f-a801-0a25df87aee2-hosts-file\") pod \"node-resolver-gmrbv\" (UID: \"c2836676-0428-4f2f-a801-0a25df87aee2\") " pod="openshift-dns/node-resolver-gmrbv"
Oct 30 19:30:46.614880 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614811    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-run-ovn-kubernetes\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.614880 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614861    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-config-volume\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:46.614880 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614861    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-service-ca\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614855    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d50ec904-b3c9-4677-9624-39cb202f65c9-system-cni-dir\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614898    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-run-ovn-kubernetes\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614883    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45ccae2d-1b91-4bbe-9598-94ad740116ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614903    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c2836676-0428-4f2f-a801-0a25df87aee2-hosts-file\") pod \"node-resolver-gmrbv\" (UID: \"c2836676-0428-4f2f-a801-0a25df87aee2\") " pod="openshift-dns/node-resolver-gmrbv"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614908    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614937    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-textfile\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614954    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53bd98ea-ea87-484a-b7b1-8c9070fc9b25-host\") pod \"node-ca-h5m52\" (UID: \"53bd98ea-ea87-484a-b7b1-8c9070fc9b25\") " pod="openshift-image-registry/node-ca-h5m52"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614883    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-metrics-tls\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.614970    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0d19f3a-000a-4911-96dd-cbf24e784627-tmp\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615008    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-textfile\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615017    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008928f8-b2ce-4f12-9681-21136d802ff6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615009    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53bd98ea-ea87-484a-b7b1-8c9070fc9b25-host\") pod \"node-ca-h5m52\" (UID: \"53bd98ea-ea87-484a-b7b1-8c9070fc9b25\") " pod="openshift-image-registry/node-ca-h5m52"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615055    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-run-netns\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615078    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45ccae2d-1b91-4bbe-9598-94ad740116ec-data-volume\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615093    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0d19f3a-000a-4911-96dd-cbf24e784627-tmp\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615103    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-oauth-serving-cert\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615149    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45ccae2d-1b91-4bbe-9598-94ad740116ec-data-volume\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615172    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-run-netns\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615180    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-trusted-ca-bundle\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615226    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-etc-kubernetes\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615274    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d50ec904-b3c9-4677-9624-39cb202f65c9-system-cni-dir\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615281    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-oauth-serving-cert\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:46.615286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615286    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-tls\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615306    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008928f8-b2ce-4f12-9681-21136d802ff6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615312    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-etc-kubernetes\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615316    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5rj9\" (UniqueName: \"kubernetes.io/projected/e0d19f3a-000a-4911-96dd-cbf24e784627-kube-api-access-w5rj9\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615366    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-config\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615393    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45ccae2d-1b91-4bbe-9598-94ad740116ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615417    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-sysctl-d\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615440    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-systemd\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615463    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-run-systemd\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615492    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-registration-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615502    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-systemd\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615517    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-tls\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615523    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615541    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-config\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615549    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-run-systemd\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615559    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e9972f4-8918-41a0-8ae4-b2014d3897ca-tmp\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615578    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-service-ca\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615597    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8bxp\" (UniqueName: \"kubernetes.io/projected/008928f8-b2ce-4f12-9681-21136d802ff6-kube-api-access-j8bxp\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615598    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-registration-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615614    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-systemd-units\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615716    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615772    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-systemd-units\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615776    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-service-ca\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615831    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e9972f4-8918-41a0-8ae4-b2014d3897ca-tmp\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615923    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-cni-netd\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616056    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-trusted-ca-bundle\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615636    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-cni-netd\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616096    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-metrics-client-ca\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616115    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45ccae2d-1b91-4bbe-9598-94ad740116ec-crio-socket\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616130    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-sys\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616145    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-socket-dir-parent\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.615624    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-sysctl-d\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616165    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-run-k8s-cni-cncf-io\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.616238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616208    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-kubernetes\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616289    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-metrics-client-ca\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616315    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-socket-dir-parent\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616348    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-node-log\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616354    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45ccae2d-1b91-4bbe-9598-94ad740116ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616357    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-kubernetes\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616383    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-run-k8s-cni-cncf-io\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616404    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616421    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-node-log\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616450    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-host\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616471    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d50ec904-b3c9-4677-9624-39cb202f65c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616485    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5sd9\" (UniqueName: \"kubernetes.io/projected/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-kube-api-access-q5sd9\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616495    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-host\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616532    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d50ec904-b3c9-4677-9624-39cb202f65c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616582    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/008928f8-b2ce-4f12-9681-21136d802ff6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616602    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-cni-dir\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616655    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-cni-dir\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616673    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-sys\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616689    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-run-ovn\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616669    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616713    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a341b7ef-207b-4443-b01e-9fcbc2ac0075-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ldlt7\" (UID: \"a341b7ef-207b-4443-b01e-9fcbc2ac0075\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616718    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-sys\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616740    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-oauth-config\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616743    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-run-ovn\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616740    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/008928f8-b2ce-4f12-9681-21136d802ff6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616783    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-oauth-serving-cert\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616838    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616896    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-oauth-config\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616932    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616934    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-var-lib-kubelet\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616963    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-run\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616981    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-run-openvswitch\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616982    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-oauth-serving-cert\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616994    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a341b7ef-207b-4443-b01e-9fcbc2ac0075-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ldlt7\" (UID: \"a341b7ef-207b-4443-b01e-9fcbc2ac0075\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.616996    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-accelerators-collector-config\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617017    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-run-openvswitch\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617044    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-var-lib-kubelet\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617050    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-run\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617064    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-var-lib-cni-multus\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617085    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617214    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-etc-openvswitch\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617103    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-etc-openvswitch\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617138    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-host-var-lib-cni-multus\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617469    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-socket-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617501    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-serving-cert\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617522    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-tls\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.617556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617553    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-var-lib-openvswitch\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617612    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-var-lib-openvswitch\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617638    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617649    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-socket-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617707    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-accelerators-collector-config\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617724    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d50ec904-b3c9-4677-9624-39cb202f65c9-cnibin\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617775    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d50ec904-b3c9-4677-9624-39cb202f65c9-cnibin\") pod \"multus-additional-cni-plugins-qmqjk\" (UID: \"d50ec904-b3c9-4677-9624-39cb202f65c9\") " pod="openshift-multus/multus-additional-cni-plugins-qmqjk"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617786    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/008928f8-b2ce-4f12-9681-21136d802ff6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617835    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-system-cni-dir\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617869    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-lib-modules\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617872    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-serving-cert\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617892    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-tls\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.617957    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-system-cni-dir\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618003    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-lib-modules\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618067    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-sys-fs\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618103    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxxqw\" (UniqueName: \"kubernetes.io/projected/0de2915a-789c-4acc-84a6-c61987498b62-kube-api-access-nxxqw\") pod \"ingress-canary-2c2l4\" (UID: \"0de2915a-789c-4acc-84a6-c61987498b62\") " pod="openshift-ingress-canary/ingress-canary-2c2l4"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618133    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-grpc-tls\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618161    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d895r\" (UniqueName: \"kubernetes.io/projected/a341b7ef-207b-4443-b01e-9fcbc2ac0075-kube-api-access-d895r\") pod \"monitoring-plugin-77f4f76594-ldlt7\" (UID: \"a341b7ef-207b-4443-b01e-9fcbc2ac0075\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618185    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-serving-cert\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618194    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/008928f8-b2ce-4f12-9681-21136d802ff6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618212    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-config\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618242    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-sysctl-conf\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618270    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-cnibin\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618275    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-sys-fs\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618300    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8013e357-0fa2-4360-924b-18cbfe588b5c-dbus\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618326    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-kubelet\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618359    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5sb5\" (UniqueName: \"kubernetes.io/projected/45ccae2d-1b91-4bbe-9598-94ad740116ec-kube-api-access-d5sb5\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618389    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k7xh\" (UniqueName: \"kubernetes.io/projected/c112dbe2-b2e4-4762-8123-b9f161bda84a-kube-api-access-6k7xh\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618417    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0de2915a-789c-4acc-84a6-c61987498b62-cert\") pod \"ingress-canary-2c2l4\" (UID: \"0de2915a-789c-4acc-84a6-c61987498b62\") " pod="openshift-ingress-canary/ingress-canary-2c2l4"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618442    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618472    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-oauth-config\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618511    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4btm\" (UniqueName: \"kubernetes.io/projected/3e9972f4-8918-41a0-8ae4-b2014d3897ca-kube-api-access-b4btm\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618543    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-hostroot\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618561    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8013e357-0fa2-4360-924b-18cbfe588b5c-dbus\") pod \"global-pull-secret-syncer-d2hv2\" (UID: \"8013e357-0fa2-4360-924b-18cbfe588b5c\") " pod="kube-system/global-pull-secret-syncer-d2hv2"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618571    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618620    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-device-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618630    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618647    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-tmp-dir\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618676    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c112dbe2-b2e4-4762-8123-b9f161bda84a-tmp\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618683    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-kubelet\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.618709    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z7pj\" (UniqueName: \"kubernetes.io/projected/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-kube-api-access-7z7pj\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.619320    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-serving-cert\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.619388    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0de2915a-789c-4acc-84a6-c61987498b62-cert\") pod \"ingress-canary-2c2l4\" (UID: \"0de2915a-789c-4acc-84a6-c61987498b62\") " pod="openshift-ingress-canary/ingress-canary-2c2l4"
Oct 30 19:30:46.619701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.619586    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.619765    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-oauth-config\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.619968    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-config\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620059    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-hostroot\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620163    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-sysctl-conf\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620225    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-cnibin\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620275    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68ff550a-daae-421b-9dd6-e9b1f33585d1-device-dir\") pod \"aws-ebs-csi-driver-node-8rjhn\" (UID: \"68ff550a-daae-421b-9dd6-e9b1f33585d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rjhn"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620314    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0d19f3a-000a-4911-96dd-cbf24e784627-secret-grpc-tls\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620350    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-tmp-dir\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620357    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d19f3a-000a-4911-96dd-cbf24e784627-metrics-client-ca\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620419    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c112dbe2-b2e4-4762-8123-b9f161bda84a-tmp\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620454    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2ac53be-96b4-4976-9581-b38cc97e1eb0-host-slash\") pod \"iptables-alerter-5n649\" (UID: \"e2ac53be-96b4-4976-9581-b38cc97e1eb0\") " pod="openshift-network-operator/iptables-alerter-5n649"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620484    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-sysconfig\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620517    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-modprobe-d\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620541    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-log-socket\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620572    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-root\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620601    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-conf-dir\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620628    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-var-lib-kubelet\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620655    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-slash\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620677    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-run-netns\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620761    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-run-netns\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620951    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d19f3a-000a-4911-96dd-cbf24e784627-metrics-client-ca\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.620999    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2ac53be-96b4-4976-9581-b38cc97e1eb0-host-slash\") pod \"iptables-alerter-5n649\" (UID: \"e2ac53be-96b4-4976-9581-b38cc97e1eb0\") " pod="openshift-network-operator/iptables-alerter-5n649"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.621229    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-var-lib-kubelet\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.621231    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-host-slash\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.621251    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-modprobe-d\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.621272    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57-multus-conf-dir\") pod \"multus-d6gqm\" (UID: \"9df58e7f-8f5f-4e3a-af1d-7e29e4b58c57\") " pod="openshift-multus/multus-d6gqm"
Oct 30 19:30:46.621643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.621297    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de8ddc3f-5d23-4cff-9f2e-9167700ff109-log-socket\") pod \"ovnkube-node-gjswr\" (UID: \"de8ddc3f-5d23-4cff-9f2e-9167700ff109\") " pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:46.622678 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.622420    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e0a98773-9cb1-4637-9188-52094d47dceb-etc-sysconfig\") pod \"tuned-bkdq5\" (UID: \"e0a98773-9cb1-4637-9188-52094d47dceb\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdq5"
Oct 30 19:30:46.625873 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.625854    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8bxp\" (UniqueName: \"kubernetes.io/projected/008928f8-b2ce-4f12-9681-21136d802ff6-kube-api-access-j8bxp\") pod \"prometheus-operator-9cd89b74c-9s95l\" (UID: \"008928f8-b2ce-4f12-9681-21136d802ff6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-9s95l"
Oct 30 19:30:46.625985 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.625936    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5rj9\" (UniqueName: \"kubernetes.io/projected/e0d19f3a-000a-4911-96dd-cbf24e784627-kube-api-access-w5rj9\") pod \"thanos-querier-67c75b6564-g8dq2\" (UID: \"e0d19f3a-000a-4911-96dd-cbf24e784627\") " pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:46.626130 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.626112    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5sd9\" (UniqueName: \"kubernetes.io/projected/c4e933e1-4e6d-4a26-bb7d-68f366e02ac5-kube-api-access-q5sd9\") pod \"dns-default-dkkwh\" (UID: \"c4e933e1-4e6d-4a26-bb7d-68f366e02ac5\") " pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:46.634486 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.634468    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxxqw\" (UniqueName: \"kubernetes.io/projected/0de2915a-789c-4acc-84a6-c61987498b62-kube-api-access-nxxqw\") pod \"ingress-canary-2c2l4\" (UID: \"0de2915a-789c-4acc-84a6-c61987498b62\") " pod="openshift-ingress-canary/ingress-canary-2c2l4"
Oct 30 19:30:46.640108 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.640089    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z7pj\" (UniqueName: \"kubernetes.io/projected/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-kube-api-access-7z7pj\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.640202 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.640132    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5sb5\" (UniqueName: \"kubernetes.io/projected/45ccae2d-1b91-4bbe-9598-94ad740116ec-kube-api-access-d5sb5\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:46.640409 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.640391    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4btm\" (UniqueName: \"kubernetes.io/projected/3e9972f4-8918-41a0-8ae4-b2014d3897ca-kube-api-access-b4btm\") pod \"console-854b9d69db-nvllp\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") " pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:30:46.640503 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.640487    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d895r\" (UniqueName: \"kubernetes.io/projected/a341b7ef-207b-4443-b01e-9fcbc2ac0075-kube-api-access-d895r\") pod \"monitoring-plugin-77f4f76594-ldlt7\" (UID: \"a341b7ef-207b-4443-b01e-9fcbc2ac0075\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7"
Oct 30 19:30:46.640628 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.640609    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k7xh\" (UniqueName: \"kubernetes.io/projected/c112dbe2-b2e4-4762-8123-b9f161bda84a-kube-api-access-6k7xh\") pod \"console-698967b46-ghthb\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") " pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:46.721178 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.721147    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-root\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.721178 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.721182    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-wtmp\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.721360 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.721214    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45ccae2d-1b91-4bbe-9598-94ad740116ec-crio-socket\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:46.721360 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.721236    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-sys\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.721360 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.721248    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-root\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.721360 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.721276    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-node-exporter-wtmp\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.721360 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.721320    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf-sys\") pod \"node-exporter-rb876\" (UID: \"0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf\") " pod="openshift-monitoring/node-exporter-rb876"
Oct 30 19:30:46.721360 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:46.721325    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45ccae2d-1b91-4bbe-9598-94ad740116ec-crio-socket\") pod \"insights-runtime-extractor-rkc6x\" (UID: \"45ccae2d-1b91-4bbe-9598-94ad740116ec\") " pod="openshift-insights/insights-runtime-extractor-rkc6x"
Oct 30 19:30:47.666091 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:47.666057    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dkkwh"
Oct 30 19:30:48.311167 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:48.311141    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:48.623022 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:48.622949    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-67c75b6564-g8dq2"
Oct 30 19:30:51.179384 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:51.179330    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:30:53.906398 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:53.906369    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:30:53.906818 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:53.906606    5408 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:30:54.480967 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:54.480934    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:30:55.228188 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:55.228156    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ldlt7"
Oct 30 19:30:58.644218 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.644107    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5f8b7447-wnrwb"]
Oct 30 19:30:58.646985 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.646963    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.653510 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.653472    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 19:30:58.659968 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.659942    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 19:30:58.663767 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.661998    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f8b7447-wnrwb"]
Oct 30 19:30:58.699573 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.699536    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8de260aa-48d7-4d43-9c07-458a5d3db82d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.699573 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.699576    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8de260aa-48d7-4d43-9c07-458a5d3db82d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.699790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.699593    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b9vs\" (UniqueName: \"kubernetes.io/projected/8de260aa-48d7-4d43-9c07-458a5d3db82d-kube-api-access-5b9vs\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.699790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.699610    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8de260aa-48d7-4d43-9c07-458a5d3db82d-telemeter-client-tls\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.699790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.699727    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8de260aa-48d7-4d43-9c07-458a5d3db82d-federate-client-tls\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.699790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.699766    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8de260aa-48d7-4d43-9c07-458a5d3db82d-metrics-client-ca\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.699943 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.699798    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8de260aa-48d7-4d43-9c07-458a5d3db82d-serving-certs-ca-bundle\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.699943 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.699883    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8de260aa-48d7-4d43-9c07-458a5d3db82d-secret-telemeter-client\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.746309 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.746280    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 19:30:58.746477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.746324    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 19:30:58.746477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.746387    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 19:30:58.746477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.746441    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-6b68c\""
Oct 30 19:30:58.746477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.746466    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 19:30:58.800862 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.800832    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8de260aa-48d7-4d43-9c07-458a5d3db82d-secret-telemeter-client\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.801059 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.800881    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8de260aa-48d7-4d43-9c07-458a5d3db82d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.801059 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.800962    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8de260aa-48d7-4d43-9c07-458a5d3db82d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.801059 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.800994    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b9vs\" (UniqueName: \"kubernetes.io/projected/8de260aa-48d7-4d43-9c07-458a5d3db82d-kube-api-access-5b9vs\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.801059 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.801023    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8de260aa-48d7-4d43-9c07-458a5d3db82d-telemeter-client-tls\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.801232 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.801082    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8de260aa-48d7-4d43-9c07-458a5d3db82d-federate-client-tls\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.801232 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.801106    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8de260aa-48d7-4d43-9c07-458a5d3db82d-metrics-client-ca\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.801232 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.801162    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8de260aa-48d7-4d43-9c07-458a5d3db82d-serving-certs-ca-bundle\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.801399 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.801339    5408 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:30:58.801993 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.801965    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8de260aa-48d7-4d43-9c07-458a5d3db82d-serving-certs-ca-bundle\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.801993 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.801977    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8de260aa-48d7-4d43-9c07-458a5d3db82d-metrics-client-ca\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.802206 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.802169    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8de260aa-48d7-4d43-9c07-458a5d3db82d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.803785 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.803764    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8de260aa-48d7-4d43-9c07-458a5d3db82d-telemeter-client-tls\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.803893 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.803871    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8de260aa-48d7-4d43-9c07-458a5d3db82d-secret-telemeter-client\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.803997 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.803985    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8de260aa-48d7-4d43-9c07-458a5d3db82d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.804174 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.804156    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8de260aa-48d7-4d43-9c07-458a5d3db82d-federate-client-tls\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.814063 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.814017    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b9vs\" (UniqueName: \"kubernetes.io/projected/8de260aa-48d7-4d43-9c07-458a5d3db82d-kube-api-access-5b9vs\") pod \"telemeter-client-5f8b7447-wnrwb\" (UID: \"8de260aa-48d7-4d43-9c07-458a5d3db82d\") " pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:58.969838 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:58.969762    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb"
Oct 30 19:30:59.250808 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:59.250783    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f8b7447-wnrwb"]
Oct 30 19:30:59.254077 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:30:59.254048    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de260aa_48d7_4d43_9c07_458a5d3db82d.slice/crio-d7f8393cf086e96e9636da0bfe27c95bceb1f08d7c5b81450819e1687c6a3fe7 WatchSource:0}: Error finding container d7f8393cf086e96e9636da0bfe27c95bceb1f08d7c5b81450819e1687c6a3fe7: Status 404 returned error can't find the container with id d7f8393cf086e96e9636da0bfe27c95bceb1f08d7c5b81450819e1687c6a3fe7
Oct 30 19:30:59.256191 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:59.256171    5408 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:30:59.633414 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:30:59.633340    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb" event={"ID":"8de260aa-48d7-4d43-9c07-458a5d3db82d","Type":"ContainerStarted","Data":"d7f8393cf086e96e9636da0bfe27c95bceb1f08d7c5b81450819e1687c6a3fe7"}
Oct 30 19:31:01.642384 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:01.642343    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb" event={"ID":"8de260aa-48d7-4d43-9c07-458a5d3db82d","Type":"ContainerStarted","Data":"9fc8fcf39b40802781601904838387748fa122573c31b3194e39c9185e1e8df1"}
Oct 30 19:31:02.647978 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.647942    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb" event={"ID":"8de260aa-48d7-4d43-9c07-458a5d3db82d","Type":"ContainerStarted","Data":"8e5dc2618872b9267e78be475580918b8cdfd097e7fbd2216e6a1bd3ada23b4c"}
Oct 30 19:31:02.648369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.647996    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb" event={"ID":"8de260aa-48d7-4d43-9c07-458a5d3db82d","Type":"ContainerStarted","Data":"d5530b5526bf2c3e70f86b88be39e356389adc24fe09e9d6c89924cd89d126f3"}
Oct 30 19:31:02.677390 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.677346    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5f8b7447-wnrwb" podStartSLOduration=2.014501279 podStartE2EDuration="4.67732814s" totalImagesPullingTime="2.662826861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:02.675655588 +0000 UTC m=+17.259925206" watchObservedRunningTime="2026-10-30 19:31:02.67732814 +0000 UTC m=+17.261597753"
Oct 30 19:31:02.774218 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.774191    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-859849b86d-cjrrt"]
Oct 30 19:31:02.776248 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.776232    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.791221 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.790452    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-859849b86d-cjrrt"]
Oct 30 19:31:02.834262 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.834234    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-oauth-serving-cert\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.834409 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.834268    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbcg9\" (UniqueName: \"kubernetes.io/projected/17cb5e75-555d-42fd-9d5b-7fbe92793eda-kube-api-access-pbcg9\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.834409 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.834295    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-service-ca\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.834409 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.834362    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-trusted-ca-bundle\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.834563 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.834441    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-serving-cert\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.834563 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.834544    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-oauth-config\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.834672 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.834582    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-config\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.834672 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.834611    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17cb5e75-555d-42fd-9d5b-7fbe92793eda-tmp\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.936012 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.935942    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-oauth-serving-cert\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.936012 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.935980    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbcg9\" (UniqueName: \"kubernetes.io/projected/17cb5e75-555d-42fd-9d5b-7fbe92793eda-kube-api-access-pbcg9\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.936012 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.935999    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-service-ca\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.936271 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.936133    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-trusted-ca-bundle\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.936271 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.936169    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-serving-cert\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.936271 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.936220    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-oauth-config\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.936271 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.936244    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-config\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.936271 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.936262    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17cb5e75-555d-42fd-9d5b-7fbe92793eda-tmp\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.936710 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.936679    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17cb5e75-555d-42fd-9d5b-7fbe92793eda-tmp\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.936822 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.936739    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-oauth-serving-cert\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.936822 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.936792    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-service-ca\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.936915 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.936896    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-config\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.937092 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.937075    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-trusted-ca-bundle\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.939179 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.939158    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-oauth-config\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.939278 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.939261    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-serving-cert\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:02.948972 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:02.948950    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbcg9\" (UniqueName: \"kubernetes.io/projected/17cb5e75-555d-42fd-9d5b-7fbe92793eda-kube-api-access-pbcg9\") pod \"console-859849b86d-cjrrt\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") " pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:03.086231 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:03.086189    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:03.358980 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:03.358945    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-859849b86d-cjrrt"]
Oct 30 19:31:03.362991 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:31:03.362968    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17cb5e75_555d_42fd_9d5b_7fbe92793eda.slice/crio-51d94fb6760fc83b5febe45a3b9ee622d29009af9912c12e663447c6c66fb93b WatchSource:0}: Error finding container 51d94fb6760fc83b5febe45a3b9ee622d29009af9912c12e663447c6c66fb93b: Status 404 returned error can't find the container with id 51d94fb6760fc83b5febe45a3b9ee622d29009af9912c12e663447c6c66fb93b
Oct 30 19:31:03.654917 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:03.654842    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-859849b86d-cjrrt" event={"ID":"17cb5e75-555d-42fd-9d5b-7fbe92793eda","Type":"ContainerStarted","Data":"120b8043b046826a99cd6c3a7c6a0ce1ae5f78340e13caf59ba7db751105e3bb"}
Oct 30 19:31:03.654917 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:03.654905    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-859849b86d-cjrrt" event={"ID":"17cb5e75-555d-42fd-9d5b-7fbe92793eda","Type":"ContainerStarted","Data":"51d94fb6760fc83b5febe45a3b9ee622d29009af9912c12e663447c6c66fb93b"}
Oct 30 19:31:03.680397 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:03.679899    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-859849b86d-cjrrt" podStartSLOduration=1.679882295 podStartE2EDuration="1.679882295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:03.679370387 +0000 UTC m=+18.263640018" watchObservedRunningTime="2026-10-30 19:31:03.679882295 +0000 UTC m=+18.264151913"
Oct 30 19:31:09.658542 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:31:09.658508    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:31:11.518365 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:11.518313    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-854b9d69db-nvllp" podUID="3e9972f4-8918-41a0-8ae4-b2014d3897ca" containerName="console" containerID="cri-o://48e72b4766f1ddcf4e665db835b405128feeabc1771724d7fca3cc955561de47" gracePeriod=15
Oct 30 19:31:11.690176 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:11.690143    5408 generic.go:356] "Generic (PLEG): container finished" podID="3e9972f4-8918-41a0-8ae4-b2014d3897ca" containerID="48e72b4766f1ddcf4e665db835b405128feeabc1771724d7fca3cc955561de47" exitCode=0
Oct 30 19:31:11.690303 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:11.690214    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854b9d69db-nvllp" event={"ID":"3e9972f4-8918-41a0-8ae4-b2014d3897ca","Type":"ContainerDied","Data":"48e72b4766f1ddcf4e665db835b405128feeabc1771724d7fca3cc955561de47"}
Oct 30 19:31:11.752433 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:11.752412    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:31:11.904940 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:11.904870    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-config\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-config\") pod \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") "
Oct 30 19:31:11.905101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:11.904943    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-oauth-config\") pod \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") "
Oct 30 19:31:11.905101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:11.905004    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-serving-cert\") pod \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") "
Oct 30 19:31:11.905101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:11.905076    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e9972f4-8918-41a0-8ae4-b2014d3897ca-kube-api-access-b4btm\" (UniqueName: \"kubernetes.io/projected/3e9972f4-8918-41a0-8ae4-b2014d3897ca-kube-api-access-b4btm\") pod \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") "
Oct 30 19:31:11.905276 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:11.905127    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-service-ca\") pod \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") "
Oct 30 19:31:11.905276 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:11.905165    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-oauth-serving-cert\") pod \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") "
Oct 30 19:31:11.905276 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:11.905210    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9972f4-8918-41a0-8ae4-b2014d3897ca-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e9972f4-8918-41a0-8ae4-b2014d3897ca-tmp\") pod \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\" (UID: \"3e9972f4-8918-41a0-8ae4-b2014d3897ca\") "
Oct 30 19:31:11.905472 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:11.905449    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-config" pod "3e9972f4-8918-41a0-8ae4-b2014d3897ca" (UID: "3e9972f4-8918-41a0-8ae4-b2014d3897ca"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:11.905602 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:11.905578    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9972f4-8918-41a0-8ae4-b2014d3897ca-tmp" pod "3e9972f4-8918-41a0-8ae4-b2014d3897ca" (UID: "3e9972f4-8918-41a0-8ae4-b2014d3897ca"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:11.905602 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:11.905588    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-service-ca" pod "3e9972f4-8918-41a0-8ae4-b2014d3897ca" (UID: "3e9972f4-8918-41a0-8ae4-b2014d3897ca"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:11.905602 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:11.905609    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-oauth-serving-cert" pod "3e9972f4-8918-41a0-8ae4-b2014d3897ca" (UID: "3e9972f4-8918-41a0-8ae4-b2014d3897ca"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:11.907359 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:11.907331    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9972f4-8918-41a0-8ae4-b2014d3897ca-kube-api-access-b4btm" pod "3e9972f4-8918-41a0-8ae4-b2014d3897ca" (UID: "3e9972f4-8918-41a0-8ae4-b2014d3897ca"). InnerVolumeSpecName "kube-api-access-b4btm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:11.907359 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:11.907348    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-oauth-config" pod "3e9972f4-8918-41a0-8ae4-b2014d3897ca" (UID: "3e9972f4-8918-41a0-8ae4-b2014d3897ca"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:11.907472 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:11.907408    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-serving-cert" pod "3e9972f4-8918-41a0-8ae4-b2014d3897ca" (UID: "3e9972f4-8918-41a0-8ae4-b2014d3897ca"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:12.006145 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:12.006112    5408 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-serving-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:12.006145 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:12.006141    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4btm\" (UniqueName: \"kubernetes.io/projected/3e9972f4-8918-41a0-8ae4-b2014d3897ca-kube-api-access-b4btm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:12.006315 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:12.006152    5408 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:12.006315 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:12.006161    5408 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-oauth-serving-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:12.006315 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:12.006170    5408 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e9972f4-8918-41a0-8ae4-b2014d3897ca-tmp\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:12.006315 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:12.006178    5408 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-config\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:12.006315 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:12.006188    5408 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3e9972f4-8918-41a0-8ae4-b2014d3897ca-console-oauth-config\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:12.695162 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:12.695130    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854b9d69db-nvllp"
Oct 30 19:31:12.695552 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:12.695130    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854b9d69db-nvllp" event={"ID":"3e9972f4-8918-41a0-8ae4-b2014d3897ca","Type":"ContainerDied","Data":"e59da0f23b582ab57c0b73f7043466ba70b4e7e2b61094e81723590699ea17f1"}
Oct 30 19:31:12.695552 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:12.695252    5408 scope.go:122] "RemoveContainer" containerID="48e72b4766f1ddcf4e665db835b405128feeabc1771724d7fca3cc955561de47"
Oct 30 19:31:12.720310 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:12.720284    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-854b9d69db-nvllp"]
Oct 30 19:31:12.724538 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:12.724513    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-854b9d69db-nvllp"]
Oct 30 19:31:13.087043 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:13.087000    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:13.087212 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:13.087109    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:13.091703 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:13.091683    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:13.528688 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:13.528651    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e9972f4-8918-41a0-8ae4-b2014d3897ca" path="/var/lib/kubelet/pods/3e9972f4-8918-41a0-8ae4-b2014d3897ca/volumes"
Oct 30 19:31:13.703638 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:13.703615    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:31:13.755710 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:13.755678    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-698967b46-ghthb"]
Oct 30 19:31:15.485272 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:15.485241    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kspfd"
Oct 30 19:31:15.923234 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:15.923203    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gjswr"
Oct 30 19:31:16.351385 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:16.351357    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2c2l4_0de2915a-789c-4acc-84a6-c61987498b62/serve-healthcheck-canary/0.log"
Oct 30 19:31:21.182238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:21.182210    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vwkv5"
Oct 30 19:31:26.978833 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:26.978802    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-857b84547b-9wx6x"]
Oct 30 19:31:26.979226 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:26.979098    5408 cpu_manager.go:414] "containerMap: removing container" podUID="3e9972f4-8918-41a0-8ae4-b2014d3897ca" containerName="console"
Oct 30 19:31:26.979226 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:26.979109    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e9972f4-8918-41a0-8ae4-b2014d3897ca" containerName="console"
Oct 30 19:31:26.979226 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:26.979156    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e9972f4-8918-41a0-8ae4-b2014d3897ca" containerName="console"
Oct 30 19:31:26.980845 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:26.980828    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:26.993053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:26.993017    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857b84547b-9wx6x"]
Oct 30 19:31:27.113411 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.113373    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9af4bc7e-b468-47a8-baee-d0c293f6f728-oauth-serving-cert\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.113584 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.113421    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9af4bc7e-b468-47a8-baee-d0c293f6f728-console-config\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.113584 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.113510    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9af4bc7e-b468-47a8-baee-d0c293f6f728-console-oauth-config\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.113584 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.113581    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9af4bc7e-b468-47a8-baee-d0c293f6f728-console-serving-cert\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.113684 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.113599    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af4bc7e-b468-47a8-baee-d0c293f6f728-trusted-ca-bundle\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.113721 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.113691    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9af4bc7e-b468-47a8-baee-d0c293f6f728-tmp\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.113754 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.113723    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9af4bc7e-b468-47a8-baee-d0c293f6f728-service-ca\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.113786 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.113756    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4rdj\" (UniqueName: \"kubernetes.io/projected/9af4bc7e-b468-47a8-baee-d0c293f6f728-kube-api-access-l4rdj\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.214152 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.214122    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9af4bc7e-b468-47a8-baee-d0c293f6f728-console-oauth-config\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.214152 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.214157    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9af4bc7e-b468-47a8-baee-d0c293f6f728-console-serving-cert\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.214415 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.214176    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af4bc7e-b468-47a8-baee-d0c293f6f728-trusted-ca-bundle\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.214415 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.214302    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9af4bc7e-b468-47a8-baee-d0c293f6f728-tmp\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.214415 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.214337    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9af4bc7e-b468-47a8-baee-d0c293f6f728-service-ca\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.214415 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.214384    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4rdj\" (UniqueName: \"kubernetes.io/projected/9af4bc7e-b468-47a8-baee-d0c293f6f728-kube-api-access-l4rdj\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.214628 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.214432    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9af4bc7e-b468-47a8-baee-d0c293f6f728-oauth-serving-cert\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.214628 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.214471    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9af4bc7e-b468-47a8-baee-d0c293f6f728-console-config\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.214741 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.214697    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9af4bc7e-b468-47a8-baee-d0c293f6f728-tmp\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.215050 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.215012    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af4bc7e-b468-47a8-baee-d0c293f6f728-trusted-ca-bundle\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.215219 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.215192    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9af4bc7e-b468-47a8-baee-d0c293f6f728-service-ca\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.215288 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.215218    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9af4bc7e-b468-47a8-baee-d0c293f6f728-console-config\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.215348 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.215304    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9af4bc7e-b468-47a8-baee-d0c293f6f728-oauth-serving-cert\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.216611 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.216589    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9af4bc7e-b468-47a8-baee-d0c293f6f728-console-oauth-config\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.216731 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.216715    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9af4bc7e-b468-47a8-baee-d0c293f6f728-console-serving-cert\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.227301 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.227279    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4rdj\" (UniqueName: \"kubernetes.io/projected/9af4bc7e-b468-47a8-baee-d0c293f6f728-kube-api-access-l4rdj\") pod \"console-857b84547b-9wx6x\" (UID: \"9af4bc7e-b468-47a8-baee-d0c293f6f728\") " pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.290914 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.290877    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:27.568267 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.568201    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857b84547b-9wx6x"]
Oct 30 19:31:27.571549 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:31:27.571514    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af4bc7e_b468_47a8_baee_d0c293f6f728.slice/crio-e96cf25dc9e375d2dcef251db1a31fefdd005f6906ab22dcc876987d42fd063f WatchSource:0}: Error finding container e96cf25dc9e375d2dcef251db1a31fefdd005f6906ab22dcc876987d42fd063f: Status 404 returned error can't find the container with id e96cf25dc9e375d2dcef251db1a31fefdd005f6906ab22dcc876987d42fd063f
Oct 30 19:31:27.748018 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.747979    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857b84547b-9wx6x" event={"ID":"9af4bc7e-b468-47a8-baee-d0c293f6f728","Type":"ContainerStarted","Data":"23b5b5b3329102f0679fc766028bd601ab064cfca9acc176ba124894e5885119"}
Oct 30 19:31:27.748206 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.748092    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857b84547b-9wx6x" event={"ID":"9af4bc7e-b468-47a8-baee-d0c293f6f728","Type":"ContainerStarted","Data":"e96cf25dc9e375d2dcef251db1a31fefdd005f6906ab22dcc876987d42fd063f"}
Oct 30 19:31:27.772414 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:27.772365    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-857b84547b-9wx6x" podStartSLOduration=1.772353313 podStartE2EDuration="1.772353313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:27.772004136 +0000 UTC m=+42.356273753" watchObservedRunningTime="2026-10-30 19:31:27.772353313 +0000 UTC m=+42.356622999"
Oct 30 19:31:37.291075 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:37.291019    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:37.291593 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:37.291130    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:37.295815 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:37.295796    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:37.794776 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:37.794747    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-857b84547b-9wx6x"
Oct 30 19:31:37.853090 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:37.853053    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-859849b86d-cjrrt"]
Oct 30 19:31:38.777944 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:38.777878    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-698967b46-ghthb" podUID="c112dbe2-b2e4-4762-8123-b9f161bda84a" containerName="console" containerID="cri-o://75df4c23c8e2de0673867ccc295fd57cded72f617ce7744def6f652591a3a45d" gracePeriod=15
Oct 30 19:31:39.023140 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.023114    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:31:39.102219 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.102124    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c112dbe2-b2e4-4762-8123-b9f161bda84a-kube-api-access-6k7xh\" (UniqueName: \"kubernetes.io/projected/c112dbe2-b2e4-4762-8123-b9f161bda84a-kube-api-access-6k7xh\") pod \"c112dbe2-b2e4-4762-8123-b9f161bda84a\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") "
Oct 30 19:31:39.102219 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.102173    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-oauth-serving-cert\") pod \"c112dbe2-b2e4-4762-8123-b9f161bda84a\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") "
Oct 30 19:31:39.102219 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.102220    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-service-ca\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-service-ca\") pod \"c112dbe2-b2e4-4762-8123-b9f161bda84a\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") "
Oct 30 19:31:39.102557 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.102245    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-config\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-config\") pod \"c112dbe2-b2e4-4762-8123-b9f161bda84a\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") "
Oct 30 19:31:39.102557 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.102277    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-oauth-config\") pod \"c112dbe2-b2e4-4762-8123-b9f161bda84a\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") "
Oct 30 19:31:39.102557 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.102326    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-trusted-ca-bundle\") pod \"c112dbe2-b2e4-4762-8123-b9f161bda84a\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") "
Oct 30 19:31:39.102557 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.102357    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-serving-cert\") pod \"c112dbe2-b2e4-4762-8123-b9f161bda84a\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") "
Oct 30 19:31:39.102557 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.102404    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c112dbe2-b2e4-4762-8123-b9f161bda84a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c112dbe2-b2e4-4762-8123-b9f161bda84a-tmp\") pod \"c112dbe2-b2e4-4762-8123-b9f161bda84a\" (UID: \"c112dbe2-b2e4-4762-8123-b9f161bda84a\") "
Oct 30 19:31:39.102794 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.102713    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-oauth-serving-cert" pod "c112dbe2-b2e4-4762-8123-b9f161bda84a" (UID: "c112dbe2-b2e4-4762-8123-b9f161bda84a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:39.102794 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.102740    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-service-ca" pod "c112dbe2-b2e4-4762-8123-b9f161bda84a" (UID: "c112dbe2-b2e4-4762-8123-b9f161bda84a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:39.102794 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.102723    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-config" pod "c112dbe2-b2e4-4762-8123-b9f161bda84a" (UID: "c112dbe2-b2e4-4762-8123-b9f161bda84a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:39.102929 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.102849    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c112dbe2-b2e4-4762-8123-b9f161bda84a-tmp" pod "c112dbe2-b2e4-4762-8123-b9f161bda84a" (UID: "c112dbe2-b2e4-4762-8123-b9f161bda84a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:39.103069 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.103023    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-trusted-ca-bundle" pod "c112dbe2-b2e4-4762-8123-b9f161bda84a" (UID: "c112dbe2-b2e4-4762-8123-b9f161bda84a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:39.104535 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.104508    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-oauth-config" pod "c112dbe2-b2e4-4762-8123-b9f161bda84a" (UID: "c112dbe2-b2e4-4762-8123-b9f161bda84a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:39.104618 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.104509    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c112dbe2-b2e4-4762-8123-b9f161bda84a-kube-api-access-6k7xh" pod "c112dbe2-b2e4-4762-8123-b9f161bda84a" (UID: "c112dbe2-b2e4-4762-8123-b9f161bda84a"). InnerVolumeSpecName "kube-api-access-6k7xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:39.104618 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.104565    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-serving-cert" pod "c112dbe2-b2e4-4762-8123-b9f161bda84a" (UID: "c112dbe2-b2e4-4762-8123-b9f161bda84a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:39.202937 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.202898    5408 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-oauth-config\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:39.202937 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.202933    5408 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-trusted-ca-bundle\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:39.202937 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.202943    5408 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-serving-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:39.203191 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.202953    5408 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c112dbe2-b2e4-4762-8123-b9f161bda84a-tmp\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:39.203191 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.202964    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k7xh\" (UniqueName: \"kubernetes.io/projected/c112dbe2-b2e4-4762-8123-b9f161bda84a-kube-api-access-6k7xh\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:39.203191 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.202974    5408 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-oauth-serving-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:39.203191 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.202982    5408 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:39.203191 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.202990    5408 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c112dbe2-b2e4-4762-8123-b9f161bda84a-console-config\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:39.658403 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:31:39.658363    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:31:39.798680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.798602    5408 generic.go:356] "Generic (PLEG): container finished" podID="c112dbe2-b2e4-4762-8123-b9f161bda84a" containerID="75df4c23c8e2de0673867ccc295fd57cded72f617ce7744def6f652591a3a45d" exitCode=0
Oct 30 19:31:39.798680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.798658    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-698967b46-ghthb" event={"ID":"c112dbe2-b2e4-4762-8123-b9f161bda84a","Type":"ContainerDied","Data":"75df4c23c8e2de0673867ccc295fd57cded72f617ce7744def6f652591a3a45d"}
Oct 30 19:31:39.798680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.798665    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-698967b46-ghthb"
Oct 30 19:31:39.799188 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.798712    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-698967b46-ghthb" event={"ID":"c112dbe2-b2e4-4762-8123-b9f161bda84a","Type":"ContainerDied","Data":"c326058e8f2e9a4ae0387a68eab1d87ddbdf11405269f074395b5f275417f1c6"}
Oct 30 19:31:39.799188 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.798739    5408 scope.go:122] "RemoveContainer" containerID="75df4c23c8e2de0673867ccc295fd57cded72f617ce7744def6f652591a3a45d"
Oct 30 19:31:39.806243 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.806227    5408 scope.go:122] "RemoveContainer" containerID="75df4c23c8e2de0673867ccc295fd57cded72f617ce7744def6f652591a3a45d"
Oct 30 19:31:39.806524 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:31:39.806506    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75df4c23c8e2de0673867ccc295fd57cded72f617ce7744def6f652591a3a45d\": container with ID starting with 75df4c23c8e2de0673867ccc295fd57cded72f617ce7744def6f652591a3a45d not found: ID does not exist" containerID="75df4c23c8e2de0673867ccc295fd57cded72f617ce7744def6f652591a3a45d"
Oct 30 19:31:39.806573 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.806535    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75df4c23c8e2de0673867ccc295fd57cded72f617ce7744def6f652591a3a45d"} err="failed to get container status \"75df4c23c8e2de0673867ccc295fd57cded72f617ce7744def6f652591a3a45d\": rpc error: code = NotFound desc = could not find container \"75df4c23c8e2de0673867ccc295fd57cded72f617ce7744def6f652591a3a45d\": container with ID starting with 75df4c23c8e2de0673867ccc295fd57cded72f617ce7744def6f652591a3a45d not found: ID does not exist"
Oct 30 19:31:39.817223 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.817197    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-698967b46-ghthb"]
Oct 30 19:31:39.823896 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:39.823876    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-698967b46-ghthb"]
Oct 30 19:31:41.528995 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:31:41.528952    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c112dbe2-b2e4-4762-8123-b9f161bda84a" path="/var/lib/kubelet/pods/c112dbe2-b2e4-4762-8123-b9f161bda84a/volumes"
Oct 30 19:32:02.872485 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:02.872431    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-859849b86d-cjrrt" podUID="17cb5e75-555d-42fd-9d5b-7fbe92793eda" containerName="console" containerID="cri-o://120b8043b046826a99cd6c3a7c6a0ce1ae5f78340e13caf59ba7db751105e3bb" gracePeriod=15
Oct 30 19:32:03.105411 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.105388    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:32:03.182617 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.182532    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-config\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-config\") pod \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") "
Oct 30 19:32:03.182617 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.182581    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-trusted-ca-bundle\") pod \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") "
Oct 30 19:32:03.182841 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.182625    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17cb5e75-555d-42fd-9d5b-7fbe92793eda-kube-api-access-pbcg9\" (UniqueName: \"kubernetes.io/projected/17cb5e75-555d-42fd-9d5b-7fbe92793eda-kube-api-access-pbcg9\") pod \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") "
Oct 30 19:32:03.182841 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.182661    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-service-ca\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-service-ca\") pod \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") "
Oct 30 19:32:03.182841 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.182692    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-serving-cert\") pod \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") "
Oct 30 19:32:03.182841 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.182727    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cb5e75-555d-42fd-9d5b-7fbe92793eda-tmp\" (UniqueName: \"kubernetes.io/empty-dir/17cb5e75-555d-42fd-9d5b-7fbe92793eda-tmp\") pod \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") "
Oct 30 19:32:03.182841 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.182765    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-oauth-config\") pod \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") "
Oct 30 19:32:03.182841 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.182796    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-oauth-serving-cert\") pod \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\" (UID: \"17cb5e75-555d-42fd-9d5b-7fbe92793eda\") "
Oct 30 19:32:03.183168 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.182961    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-config" pod "17cb5e75-555d-42fd-9d5b-7fbe92793eda" (UID: "17cb5e75-555d-42fd-9d5b-7fbe92793eda"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:03.183168 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.183128    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cb5e75-555d-42fd-9d5b-7fbe92793eda-tmp" pod "17cb5e75-555d-42fd-9d5b-7fbe92793eda" (UID: "17cb5e75-555d-42fd-9d5b-7fbe92793eda"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:03.183322 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.183179    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-trusted-ca-bundle" pod "17cb5e75-555d-42fd-9d5b-7fbe92793eda" (UID: "17cb5e75-555d-42fd-9d5b-7fbe92793eda"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:03.183322 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.183189    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-service-ca" pod "17cb5e75-555d-42fd-9d5b-7fbe92793eda" (UID: "17cb5e75-555d-42fd-9d5b-7fbe92793eda"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:03.183420 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.183368    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-oauth-serving-cert" pod "17cb5e75-555d-42fd-9d5b-7fbe92793eda" (UID: "17cb5e75-555d-42fd-9d5b-7fbe92793eda"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:03.184832 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.184808    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-serving-cert" pod "17cb5e75-555d-42fd-9d5b-7fbe92793eda" (UID: "17cb5e75-555d-42fd-9d5b-7fbe92793eda"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:03.184927 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.184847    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17cb5e75-555d-42fd-9d5b-7fbe92793eda-kube-api-access-pbcg9" pod "17cb5e75-555d-42fd-9d5b-7fbe92793eda" (UID: "17cb5e75-555d-42fd-9d5b-7fbe92793eda"). InnerVolumeSpecName "kube-api-access-pbcg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:03.184927 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.184862    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-oauth-config" pod "17cb5e75-555d-42fd-9d5b-7fbe92793eda" (UID: "17cb5e75-555d-42fd-9d5b-7fbe92793eda"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:03.284358 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.284323    5408 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-config\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:03.284358 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.284351    5408 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-trusted-ca-bundle\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:03.284358 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.284361    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbcg9\" (UniqueName: \"kubernetes.io/projected/17cb5e75-555d-42fd-9d5b-7fbe92793eda-kube-api-access-pbcg9\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:03.284583 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.284371    5408 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:03.284583 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.284380    5408 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-serving-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:03.284583 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.284389    5408 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17cb5e75-555d-42fd-9d5b-7fbe92793eda-tmp\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:03.284583 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.284399    5408 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17cb5e75-555d-42fd-9d5b-7fbe92793eda-console-oauth-config\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:03.284583 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.284408    5408 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17cb5e75-555d-42fd-9d5b-7fbe92793eda-oauth-serving-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:03.876974 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.876876    5408 generic.go:356] "Generic (PLEG): container finished" podID="17cb5e75-555d-42fd-9d5b-7fbe92793eda" containerID="120b8043b046826a99cd6c3a7c6a0ce1ae5f78340e13caf59ba7db751105e3bb" exitCode=0
Oct 30 19:32:03.876974 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.876934    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-859849b86d-cjrrt" event={"ID":"17cb5e75-555d-42fd-9d5b-7fbe92793eda","Type":"ContainerDied","Data":"120b8043b046826a99cd6c3a7c6a0ce1ae5f78340e13caf59ba7db751105e3bb"}
Oct 30 19:32:03.877488 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.877044    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-859849b86d-cjrrt" event={"ID":"17cb5e75-555d-42fd-9d5b-7fbe92793eda","Type":"ContainerDied","Data":"51d94fb6760fc83b5febe45a3b9ee622d29009af9912c12e663447c6c66fb93b"}
Oct 30 19:32:03.877488 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.877082    5408 scope.go:122] "RemoveContainer" containerID="120b8043b046826a99cd6c3a7c6a0ce1ae5f78340e13caf59ba7db751105e3bb"
Oct 30 19:32:03.877488 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.877267    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-859849b86d-cjrrt"
Oct 30 19:32:03.888084 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.888065    5408 scope.go:122] "RemoveContainer" containerID="120b8043b046826a99cd6c3a7c6a0ce1ae5f78340e13caf59ba7db751105e3bb"
Oct 30 19:32:03.888331 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:32:03.888313    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"120b8043b046826a99cd6c3a7c6a0ce1ae5f78340e13caf59ba7db751105e3bb\": container with ID starting with 120b8043b046826a99cd6c3a7c6a0ce1ae5f78340e13caf59ba7db751105e3bb not found: ID does not exist" containerID="120b8043b046826a99cd6c3a7c6a0ce1ae5f78340e13caf59ba7db751105e3bb"
Oct 30 19:32:03.888377 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.888341    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"120b8043b046826a99cd6c3a7c6a0ce1ae5f78340e13caf59ba7db751105e3bb"} err="failed to get container status \"120b8043b046826a99cd6c3a7c6a0ce1ae5f78340e13caf59ba7db751105e3bb\": rpc error: code = NotFound desc = could not find container \"120b8043b046826a99cd6c3a7c6a0ce1ae5f78340e13caf59ba7db751105e3bb\": container with ID starting with 120b8043b046826a99cd6c3a7c6a0ce1ae5f78340e13caf59ba7db751105e3bb not found: ID does not exist"
Oct 30 19:32:03.895691 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.895671    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-859849b86d-cjrrt"]
Oct 30 19:32:03.900359 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:03.900340    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-859849b86d-cjrrt"]
Oct 30 19:32:05.529023 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:32:05.528991    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17cb5e75-555d-42fd-9d5b-7fbe92793eda" path="/var/lib/kubelet/pods/17cb5e75-555d-42fd-9d5b-7fbe92793eda/volumes"
Oct 30 19:32:09.658177 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:32:09.658136    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:32:39.657915 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:32:39.657839    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:33:09.658396 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:33:09.658361    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:33:39.659417 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:33:39.659374    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:33:49.361142 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.361103    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56"]
Oct 30 19:33:49.361742 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.361498    5408 cpu_manager.go:414] "containerMap: removing container" podUID="17cb5e75-555d-42fd-9d5b-7fbe92793eda" containerName="console"
Oct 30 19:33:49.361742 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.361515    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17cb5e75-555d-42fd-9d5b-7fbe92793eda" containerName="console"
Oct 30 19:33:49.361742 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.361540    5408 cpu_manager.go:414] "containerMap: removing container" podUID="c112dbe2-b2e4-4762-8123-b9f161bda84a" containerName="console"
Oct 30 19:33:49.361742 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.361548    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c112dbe2-b2e4-4762-8123-b9f161bda84a" containerName="console"
Oct 30 19:33:49.361742 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.361612    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="17cb5e75-555d-42fd-9d5b-7fbe92793eda" containerName="console"
Oct 30 19:33:49.361742 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.361627    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="c112dbe2-b2e4-4762-8123-b9f161bda84a" containerName="console"
Oct 30 19:33:49.364732 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.364710    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56"
Oct 30 19:33:49.369001 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.368979    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 19:33:49.370794 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.370770    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 19:33:49.370944 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.370879    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-9db9c\""
Oct 30 19:33:49.376709 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.376687    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56"]
Oct 30 19:33:49.429768 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.429742    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56\" (UID: \"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56"
Oct 30 19:33:49.429885 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.429777    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56\" (UID: \"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56"
Oct 30 19:33:49.429885 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.429806    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb6nc\" (UniqueName: \"kubernetes.io/projected/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-kube-api-access-sb6nc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56\" (UID: \"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56"
Oct 30 19:33:49.474645 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.474615    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"]
Oct 30 19:33:49.477830 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.477812    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.482876 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.482852    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:33:49.482959 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.482879    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:33:49.483356 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.483340    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 19:33:49.483421 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.483375    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 19:33:49.483421 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.483373    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sh8sg\""
Oct 30 19:33:49.483421 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.483389    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 19:33:49.484438 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.484423    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:33:49.496756 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.496734    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"]
Oct 30 19:33:49.530904 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.530878    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.531006 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.530905    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csdvd\" (UniqueName: \"kubernetes.io/projected/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-kube-api-access-csdvd\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.531006 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.530942    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56\" (UID: \"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56"
Oct 30 19:33:49.531116 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.531001    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.531116 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.531047    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56\" (UID: \"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56"
Oct 30 19:33:49.531116 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.531066    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.531265 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.531122    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb6nc\" (UniqueName: \"kubernetes.io/projected/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-kube-api-access-sb6nc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56\" (UID: \"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56"
Oct 30 19:33:49.531265 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.531221    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.531371 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.531350    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.531430 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.531396    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.531515 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.531494    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56\" (UID: \"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56"
Oct 30 19:33:49.531569 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.531503    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56\" (UID: \"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56"
Oct 30 19:33:49.549867 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.549838    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb6nc\" (UniqueName: \"kubernetes.io/projected/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-kube-api-access-sb6nc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56\" (UID: \"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56"
Oct 30 19:33:49.632515 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.632452    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.632515 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.632486    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.632675 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.632513    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.632675 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.632542    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.632675 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.632573    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.632675 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.632637    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.632675 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.632660    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csdvd\" (UniqueName: \"kubernetes.io/projected/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-kube-api-access-csdvd\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.633195 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.633168    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.634739 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.634712    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.634922 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.634906    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.634988 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.634945    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.634988 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.634954    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.645248 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.645224    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.645349 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.645302    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csdvd\" (UniqueName: \"kubernetes.io/projected/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-kube-api-access-csdvd\") pod \"istiod-openshift-gateway-7876bfcb74-5c2dq\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.685125 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.685103    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56"
Oct 30 19:33:49.788746 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.788713    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:49.962576 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:49.962506    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56"]
Oct 30 19:33:49.965829 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:33:49.965799    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5753b0f_9151_4dcb_a6ce_fd87576fb1c3.slice/crio-2586bb18c1a27230fde4b23158d63a446c81d307155422c5208033c008b2bf82 WatchSource:0}: Error finding container 2586bb18c1a27230fde4b23158d63a446c81d307155422c5208033c008b2bf82: Status 404 returned error can't find the container with id 2586bb18c1a27230fde4b23158d63a446c81d307155422c5208033c008b2bf82
Oct 30 19:33:50.063921 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:50.063850    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"]
Oct 30 19:33:50.068262 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:33:50.068239    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46163f29_52b0_4ec1_a104_2ad8fc1d66b7.slice/crio-85268447a0c08e4550f90f1e911d6b959c14f6bc253f530be10e0414717c8dd5 WatchSource:0}: Error finding container 85268447a0c08e4550f90f1e911d6b959c14f6bc253f530be10e0414717c8dd5: Status 404 returned error can't find the container with id 85268447a0c08e4550f90f1e911d6b959c14f6bc253f530be10e0414717c8dd5
Oct 30 19:33:50.212799 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:50.212711    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56" event={"ID":"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3","Type":"ContainerStarted","Data":"2586bb18c1a27230fde4b23158d63a446c81d307155422c5208033c008b2bf82"}
Oct 30 19:33:50.213820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:50.213795    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq" event={"ID":"46163f29-52b0-4ec1-a104-2ad8fc1d66b7","Type":"ContainerStarted","Data":"85268447a0c08e4550f90f1e911d6b959c14f6bc253f530be10e0414717c8dd5"}
Oct 30 19:33:53.062695 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:53.062664    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:33:53.062949 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:53.062737    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:33:53.227775 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:53.227734    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq" event={"ID":"46163f29-52b0-4ec1-a104-2ad8fc1d66b7","Type":"ContainerStarted","Data":"36fdbe916d76651ea7824ee924ccfdc787a27e5796885dd13f67d6c729e7307c"}
Oct 30 19:33:53.227955 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:53.227809    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:33:53.255794 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:53.255745    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq" podStartSLOduration=1.263973478 podStartE2EDuration="4.255731871s" totalImagesPullingTime="2.991758393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:53.255432002 +0000 UTC m=+187.839701620" watchObservedRunningTime="2026-10-30 19:33:53.255731871 +0000 UTC m=+187.840001485"
Oct 30 19:33:54.233547 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:54.233514    5408 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-5c2dq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:33:54.233897 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:54.233564    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq" podUID="46163f29-52b0-4ec1-a104-2ad8fc1d66b7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:33:57.232980 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:33:57.232899    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:34:00.254496 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:00.254461    5408 generic.go:356] "Generic (PLEG): container finished" podID="f5753b0f-9151-4dcb-a6ce-fd87576fb1c3" containerID="a0cc469ceca4daabfb893ca9e93d2ac8d99f52abfbfac1571a51be7c84814a0a" exitCode=0
Oct 30 19:34:00.254863 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:00.254542    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56" event={"ID":"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3","Type":"ContainerDied","Data":"a0cc469ceca4daabfb893ca9e93d2ac8d99f52abfbfac1571a51be7c84814a0a"}
Oct 30 19:34:01.259724 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:01.259681    5408 generic.go:356] "Generic (PLEG): container finished" podID="f5753b0f-9151-4dcb-a6ce-fd87576fb1c3" containerID="f8c583293a88dacf2be6dde8ea09a6316925b4c628643480b405ba5a8a839d32" exitCode=0
Oct 30 19:34:01.260151 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:01.259742    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56" event={"ID":"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3","Type":"ContainerDied","Data":"f8c583293a88dacf2be6dde8ea09a6316925b4c628643480b405ba5a8a839d32"}
Oct 30 19:34:09.662739 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:34:09.662701    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:34:12.297660 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:12.297614    5408 generic.go:356] "Generic (PLEG): container finished" podID="f5753b0f-9151-4dcb-a6ce-fd87576fb1c3" containerID="4f1fd094aceb1c80b0ae9d4d255a542d547d6cfe4dee899c056d0aac18abd252" exitCode=0
Oct 30 19:34:12.298065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:12.297694    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56" event={"ID":"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3","Type":"ContainerDied","Data":"4f1fd094aceb1c80b0ae9d4d255a542d547d6cfe4dee899c056d0aac18abd252"}
Oct 30 19:34:13.417587 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:13.417557    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56"
Oct 30 19:34:13.527561 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:13.527533    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-bundle\") pod \"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3\" (UID: \"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3\") "
Oct 30 19:34:13.527725 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:13.527596    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-util\" (UniqueName: \"kubernetes.io/empty-dir/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-util\") pod \"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3\" (UID: \"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3\") "
Oct 30 19:34:13.527725 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:13.527649    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-kube-api-access-sb6nc\" (UniqueName: \"kubernetes.io/projected/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-kube-api-access-sb6nc\") pod \"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3\" (UID: \"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3\") "
Oct 30 19:34:13.528230 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:13.528204    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-bundle" pod "f5753b0f-9151-4dcb-a6ce-fd87576fb1c3" (UID: "f5753b0f-9151-4dcb-a6ce-fd87576fb1c3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:13.530006 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:13.529978    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-kube-api-access-sb6nc" pod "f5753b0f-9151-4dcb-a6ce-fd87576fb1c3" (UID: "f5753b0f-9151-4dcb-a6ce-fd87576fb1c3"). InnerVolumeSpecName "kube-api-access-sb6nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:13.532090 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:13.532063    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-util" pod "f5753b0f-9151-4dcb-a6ce-fd87576fb1c3" (UID: "f5753b0f-9151-4dcb-a6ce-fd87576fb1c3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:13.628334 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:13.628271    5408 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-util\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:13.628334 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:13.628297    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb6nc\" (UniqueName: \"kubernetes.io/projected/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-kube-api-access-sb6nc\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:13.628334 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:13.628308    5408 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5753b0f-9151-4dcb-a6ce-fd87576fb1c3-bundle\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:14.307205 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:14.307171    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56" event={"ID":"f5753b0f-9151-4dcb-a6ce-fd87576fb1c3","Type":"ContainerDied","Data":"2586bb18c1a27230fde4b23158d63a446c81d307155422c5208033c008b2bf82"}
Oct 30 19:34:14.307370 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:14.307223    5408 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2586bb18c1a27230fde4b23158d63a446c81d307155422c5208033c008b2bf82"
Oct 30 19:34:14.307370 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:14.307181    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hfp56"
Oct 30 19:34:34.867839 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.867801    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"]
Oct 30 19:34:34.870236 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.868095    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f5753b0f-9151-4dcb-a6ce-fd87576fb1c3" containerName="pull"
Oct 30 19:34:34.870236 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.868108    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5753b0f-9151-4dcb-a6ce-fd87576fb1c3" containerName="pull"
Oct 30 19:34:34.870236 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.868127    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f5753b0f-9151-4dcb-a6ce-fd87576fb1c3" containerName="util"
Oct 30 19:34:34.870236 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.868132    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5753b0f-9151-4dcb-a6ce-fd87576fb1c3" containerName="util"
Oct 30 19:34:34.870236 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.868140    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f5753b0f-9151-4dcb-a6ce-fd87576fb1c3" containerName="extract"
Oct 30 19:34:34.870236 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.868145    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5753b0f-9151-4dcb-a6ce-fd87576fb1c3" containerName="extract"
Oct 30 19:34:34.870236 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.868200    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5753b0f-9151-4dcb-a6ce-fd87576fb1c3" containerName="extract"
Oct 30 19:34:34.871322 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.871308    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:34.884139 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.884111    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn9mp\" (UniqueName: \"kubernetes.io/projected/32247571-fd12-4118-b5af-7c1b57da25ed-kube-api-access-fn9mp\") pod \"opendatahub-operator-controller-manager-85768f955d-fcxts\" (UID: \"32247571-fd12-4118-b5af-7c1b57da25ed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:34.884261 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.884159    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/32247571-fd12-4118-b5af-7c1b57da25ed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-fcxts\" (UID: \"32247571-fd12-4118-b5af-7c1b57da25ed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:34.884261 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.884239    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32247571-fd12-4118-b5af-7c1b57da25ed-tmp\") pod \"opendatahub-operator-controller-manager-85768f955d-fcxts\" (UID: \"32247571-fd12-4118-b5af-7c1b57da25ed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:34.884344 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.884314    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/32247571-fd12-4118-b5af-7c1b57da25ed-manifests\") pod \"opendatahub-operator-controller-manager-85768f955d-fcxts\" (UID: \"32247571-fd12-4118-b5af-7c1b57da25ed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:34.884383 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.884354    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32247571-fd12-4118-b5af-7c1b57da25ed-metrics-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-fcxts\" (UID: \"32247571-fd12-4118-b5af-7c1b57da25ed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:34.884441 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.884422    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32247571-fd12-4118-b5af-7c1b57da25ed-webhook-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-fcxts\" (UID: \"32247571-fd12-4118-b5af-7c1b57da25ed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:34.925190 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.925149    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"]
Oct 30 19:34:34.942404 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.942375    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 19:34:34.942571 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.942554    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-klkgs\""
Oct 30 19:34:34.942647 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.942607    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 19:34:34.942721 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.942677    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 19:34:34.947745 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.947706    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 19:34:34.947978 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.947963    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 19:34:34.985137 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.985116    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32247571-fd12-4118-b5af-7c1b57da25ed-webhook-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-fcxts\" (UID: \"32247571-fd12-4118-b5af-7c1b57da25ed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:34.985263 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.985151    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn9mp\" (UniqueName: \"kubernetes.io/projected/32247571-fd12-4118-b5af-7c1b57da25ed-kube-api-access-fn9mp\") pod \"opendatahub-operator-controller-manager-85768f955d-fcxts\" (UID: \"32247571-fd12-4118-b5af-7c1b57da25ed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:34.985263 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.985173    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/32247571-fd12-4118-b5af-7c1b57da25ed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-fcxts\" (UID: \"32247571-fd12-4118-b5af-7c1b57da25ed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:34.985349 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.985294    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32247571-fd12-4118-b5af-7c1b57da25ed-tmp\") pod \"opendatahub-operator-controller-manager-85768f955d-fcxts\" (UID: \"32247571-fd12-4118-b5af-7c1b57da25ed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:34.985404 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.985353    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/32247571-fd12-4118-b5af-7c1b57da25ed-manifests\") pod \"opendatahub-operator-controller-manager-85768f955d-fcxts\" (UID: \"32247571-fd12-4118-b5af-7c1b57da25ed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:34.985453 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.985402    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32247571-fd12-4118-b5af-7c1b57da25ed-metrics-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-fcxts\" (UID: \"32247571-fd12-4118-b5af-7c1b57da25ed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:34.985711 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.985686    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/32247571-fd12-4118-b5af-7c1b57da25ed-manifests\") pod \"opendatahub-operator-controller-manager-85768f955d-fcxts\" (UID: \"32247571-fd12-4118-b5af-7c1b57da25ed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:34.987746 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.987723    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32247571-fd12-4118-b5af-7c1b57da25ed-webhook-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-fcxts\" (UID: \"32247571-fd12-4118-b5af-7c1b57da25ed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:34.987833 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.987743    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32247571-fd12-4118-b5af-7c1b57da25ed-metrics-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-fcxts\" (UID: \"32247571-fd12-4118-b5af-7c1b57da25ed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:34.987913 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.987892    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32247571-fd12-4118-b5af-7c1b57da25ed-tmp\") pod \"opendatahub-operator-controller-manager-85768f955d-fcxts\" (UID: \"32247571-fd12-4118-b5af-7c1b57da25ed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:34.988056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.988015    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/32247571-fd12-4118-b5af-7c1b57da25ed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-fcxts\" (UID: \"32247571-fd12-4118-b5af-7c1b57da25ed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:34.998893 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:34.998868    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn9mp\" (UniqueName: \"kubernetes.io/projected/32247571-fd12-4118-b5af-7c1b57da25ed-kube-api-access-fn9mp\") pod \"opendatahub-operator-controller-manager-85768f955d-fcxts\" (UID: \"32247571-fd12-4118-b5af-7c1b57da25ed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:35.186348 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:35.186258    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:35.491982 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:35.491951    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"]
Oct 30 19:34:35.496044 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:34:35.496013    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32247571_fd12_4118_b5af_7c1b57da25ed.slice/crio-228b708fb689d0cad63a3438461186ed0f5b98ce4c8c4749db9c37cb625c0dd0 WatchSource:0}: Error finding container 228b708fb689d0cad63a3438461186ed0f5b98ce4c8c4749db9c37cb625c0dd0: Status 404 returned error can't find the container with id 228b708fb689d0cad63a3438461186ed0f5b98ce4c8c4749db9c37cb625c0dd0
Oct 30 19:34:36.381482 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:36.381442    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts" event={"ID":"32247571-fd12-4118-b5af-7c1b57da25ed","Type":"ContainerStarted","Data":"228b708fb689d0cad63a3438461186ed0f5b98ce4c8c4749db9c37cb625c0dd0"}
Oct 30 19:34:39.393987 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:39.393909    5408 generic.go:356] "Generic (PLEG): container finished" podID="32247571-fd12-4118-b5af-7c1b57da25ed" containerID="557e94b2e21c373f147254cff717263f65383662a01e325d123e15e7ba32e478" exitCode=0
Oct 30 19:34:39.393987 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:39.393947    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts" event={"ID":"32247571-fd12-4118-b5af-7c1b57da25ed","Type":"ContainerDied","Data":"557e94b2e21c373f147254cff717263f65383662a01e325d123e15e7ba32e478"}
Oct 30 19:34:39.658237 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:34:39.658153    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:34:40.398886 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:40.398848    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts" event={"ID":"32247571-fd12-4118-b5af-7c1b57da25ed","Type":"ContainerStarted","Data":"edd03792b52fe2c64560cbaf52600e8f50cc675a53ecc6acf6696f94cab62d36"}
Oct 30 19:34:40.399321 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:40.399013    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:34:40.465669 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:40.465620    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts" podStartSLOduration=2.593528457 podStartE2EDuration="6.465606477s" totalImagesPullingTime="3.388695041s" totalInitContainerRuntime="483.382979ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:40.46376772 +0000 UTC m=+235.048037336" watchObservedRunningTime="2026-10-30 19:34:40.465606477 +0000 UTC m=+235.049876094"
Oct 30 19:34:51.404645 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:34:51.404610    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-fcxts"
Oct 30 19:35:09.659185 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:35:09.659143    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:35:10.988398 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:10.988355    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d87896589-x8jvc"]
Oct 30 19:35:10.994274 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:10.994245    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d87896589-x8jvc"
Oct 30 19:35:11.014277 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:11.014248    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d87896589-x8jvc"]
Oct 30 19:35:11.067158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:11.067117    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb-tls-certs\") pod \"kube-auth-proxy-5d87896589-x8jvc\" (UID: \"59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x8jvc"
Oct 30 19:35:11.067329 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:11.067311    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72tb8\" (UniqueName: \"kubernetes.io/projected/59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb-kube-api-access-72tb8\") pod \"kube-auth-proxy-5d87896589-x8jvc\" (UID: \"59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x8jvc"
Oct 30 19:35:11.067391 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:11.067373    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb-tmp\") pod \"kube-auth-proxy-5d87896589-x8jvc\" (UID: \"59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x8jvc"
Oct 30 19:35:11.087817 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:11.087793    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 19:35:11.088006 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:11.087989    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 19:35:11.088103 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:11.088084    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fqw8g\""
Oct 30 19:35:11.167842 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:11.167815    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb-tmp\") pod \"kube-auth-proxy-5d87896589-x8jvc\" (UID: \"59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x8jvc"
Oct 30 19:35:11.168005 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:11.167886    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb-tls-certs\") pod \"kube-auth-proxy-5d87896589-x8jvc\" (UID: \"59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x8jvc"
Oct 30 19:35:11.168005 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:11.167948    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72tb8\" (UniqueName: \"kubernetes.io/projected/59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb-kube-api-access-72tb8\") pod \"kube-auth-proxy-5d87896589-x8jvc\" (UID: \"59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x8jvc"
Oct 30 19:35:11.168152 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:35:11.168061    5408 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Oct 30 19:35:11.168217 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:35:11.168156    5408 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb-tls-certs podName:59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb nodeName:}" failed. No retries permitted until 2026-10-30 19:35:11.668132276 +0000 UTC m=+266.252401887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb-tls-certs") pod "kube-auth-proxy-5d87896589-x8jvc" (UID: "59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb") : secret "kube-auth-proxy-tls" not found
Oct 30 19:35:11.169985 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:11.169968    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb-tmp\") pod \"kube-auth-proxy-5d87896589-x8jvc\" (UID: \"59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x8jvc"
Oct 30 19:35:11.183548 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:11.183527    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72tb8\" (UniqueName: \"kubernetes.io/projected/59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb-kube-api-access-72tb8\") pod \"kube-auth-proxy-5d87896589-x8jvc\" (UID: \"59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x8jvc"
Oct 30 19:35:11.671146 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:11.671115    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb-tls-certs\") pod \"kube-auth-proxy-5d87896589-x8jvc\" (UID: \"59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x8jvc"
Oct 30 19:35:11.673448 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:11.673429    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb-tls-certs\") pod \"kube-auth-proxy-5d87896589-x8jvc\" (UID: \"59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x8jvc"
Oct 30 19:35:11.905822 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:11.905785    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d87896589-x8jvc"
Oct 30 19:35:12.178382 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:12.178351    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d87896589-x8jvc"]
Oct 30 19:35:12.181403 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:35:12.181374    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f8e2b6_46cc_487a_9c8b_20fb1c7f63eb.slice/crio-ecf979175eb9ab78398735d8348a8c8d2e95066c44ce9d0b351bcae573783f39 WatchSource:0}: Error finding container ecf979175eb9ab78398735d8348a8c8d2e95066c44ce9d0b351bcae573783f39: Status 404 returned error can't find the container with id ecf979175eb9ab78398735d8348a8c8d2e95066c44ce9d0b351bcae573783f39
Oct 30 19:35:12.506045 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:12.506000    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d87896589-x8jvc" event={"ID":"59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb","Type":"ContainerStarted","Data":"ecf979175eb9ab78398735d8348a8c8d2e95066c44ce9d0b351bcae573783f39"}
Oct 30 19:35:15.518476 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:15.518443    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d87896589-x8jvc" event={"ID":"59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb","Type":"ContainerStarted","Data":"113d505c5f8fc5f31d0c717fe63e033d8b10262bda35c25baddda5a2ebb5846a"}
Oct 30 19:35:15.538709 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:15.538654    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d87896589-x8jvc" podStartSLOduration=3.249283791 podStartE2EDuration="5.53863541s" totalImagesPullingTime="2.289351619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:15.537323861 +0000 UTC m=+270.121593478" watchObservedRunningTime="2026-10-30 19:35:15.53863541 +0000 UTC m=+270.122905029"
Oct 30 19:35:16.643102 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:16.643069    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-z4wjh"]
Oct 30 19:35:16.646265 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:16.646249    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z4wjh"
Oct 30 19:35:16.650461 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:16.650433    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-226ws\""
Oct 30 19:35:16.651598 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:16.651582    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 19:35:16.651704 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:16.651604    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 19:35:16.661276 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:16.661253    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-z4wjh"]
Oct 30 19:35:16.812193 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:16.812166    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69c42c28-9b66-4569-81b7-c1e9c8ff4038-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-z4wjh\" (UID: \"69c42c28-9b66-4569-81b7-c1e9c8ff4038\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z4wjh"
Oct 30 19:35:16.812373 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:16.812201    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n44jb\" (UniqueName: \"kubernetes.io/projected/69c42c28-9b66-4569-81b7-c1e9c8ff4038-kube-api-access-n44jb\") pod \"cert-manager-cainjector-5c57bbd4fc-z4wjh\" (UID: \"69c42c28-9b66-4569-81b7-c1e9c8ff4038\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z4wjh"
Oct 30 19:35:16.913489 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:16.913418    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69c42c28-9b66-4569-81b7-c1e9c8ff4038-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-z4wjh\" (UID: \"69c42c28-9b66-4569-81b7-c1e9c8ff4038\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z4wjh"
Oct 30 19:35:16.913489 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:16.913458    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n44jb\" (UniqueName: \"kubernetes.io/projected/69c42c28-9b66-4569-81b7-c1e9c8ff4038-kube-api-access-n44jb\") pod \"cert-manager-cainjector-5c57bbd4fc-z4wjh\" (UID: \"69c42c28-9b66-4569-81b7-c1e9c8ff4038\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z4wjh"
Oct 30 19:35:16.928266 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:16.928238    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69c42c28-9b66-4569-81b7-c1e9c8ff4038-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-z4wjh\" (UID: \"69c42c28-9b66-4569-81b7-c1e9c8ff4038\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z4wjh"
Oct 30 19:35:16.928369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:16.928352    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n44jb\" (UniqueName: \"kubernetes.io/projected/69c42c28-9b66-4569-81b7-c1e9c8ff4038-kube-api-access-n44jb\") pod \"cert-manager-cainjector-5c57bbd4fc-z4wjh\" (UID: \"69c42c28-9b66-4569-81b7-c1e9c8ff4038\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z4wjh"
Oct 30 19:35:16.967418 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:16.967397    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z4wjh"
Oct 30 19:35:17.252686 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:17.252656    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-z4wjh"]
Oct 30 19:35:17.255927 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:35:17.255900    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69c42c28_9b66_4569_81b7_c1e9c8ff4038.slice/crio-496451c59c85760b795a7c46971e23ff8d1b8d3633fcd134ef65314acb13759a WatchSource:0}: Error finding container 496451c59c85760b795a7c46971e23ff8d1b8d3633fcd134ef65314acb13759a: Status 404 returned error can't find the container with id 496451c59c85760b795a7c46971e23ff8d1b8d3633fcd134ef65314acb13759a
Oct 30 19:35:17.527847 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:17.527772    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z4wjh" event={"ID":"69c42c28-9b66-4569-81b7-c1e9c8ff4038","Type":"ContainerStarted","Data":"496451c59c85760b795a7c46971e23ff8d1b8d3633fcd134ef65314acb13759a"}
Oct 30 19:35:20.540228 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:20.540186    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z4wjh" event={"ID":"69c42c28-9b66-4569-81b7-c1e9c8ff4038","Type":"ContainerStarted","Data":"adc96a1bb87d4f53b3849e2a2ac5544460cd7fef8578a94ea8bbe50ce979d627"}
Oct 30 19:35:20.563103 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:35:20.563056    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z4wjh" podStartSLOduration=1.415595856 podStartE2EDuration="4.56304221s" totalImagesPullingTime="3.147446354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:20.562828622 +0000 UTC m=+275.147098242" watchObservedRunningTime="2026-10-30 19:35:20.56304221 +0000 UTC m=+275.147311819"
Oct 30 19:35:39.658557 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:35:39.658475    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:36:09.658373 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:36:09.658336    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:36:10.318328 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.318295    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx"]
Oct 30 19:36:10.321351 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.321328    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx"
Oct 30 19:36:10.326440 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.326402    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-5j97b\""
Oct 30 19:36:10.328579 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.328562    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 19:36:10.329168 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.329151    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 19:36:10.340013 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.339980    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx"]
Oct 30 19:36:10.395661 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.395632    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 19:36:10.439812 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.439786    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xv8m\" (UniqueName: \"kubernetes.io/projected/854f323b-efd5-4bac-8ffa-72bd000cfb8f-kube-api-access-6xv8m\") pod \"llmisvc-controller-manager-869d5dcc5d-thswx\" (UID: \"854f323b-efd5-4bac-8ffa-72bd000cfb8f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx"
Oct 30 19:36:10.439938 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.439838    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/854f323b-efd5-4bac-8ffa-72bd000cfb8f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-thswx\" (UID: \"854f323b-efd5-4bac-8ffa-72bd000cfb8f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx"
Oct 30 19:36:10.540631 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.540601    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/854f323b-efd5-4bac-8ffa-72bd000cfb8f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-thswx\" (UID: \"854f323b-efd5-4bac-8ffa-72bd000cfb8f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx"
Oct 30 19:36:10.540770 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.540674    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xv8m\" (UniqueName: \"kubernetes.io/projected/854f323b-efd5-4bac-8ffa-72bd000cfb8f-kube-api-access-6xv8m\") pod \"llmisvc-controller-manager-869d5dcc5d-thswx\" (UID: \"854f323b-efd5-4bac-8ffa-72bd000cfb8f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx"
Oct 30 19:36:10.542952 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.542928    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/854f323b-efd5-4bac-8ffa-72bd000cfb8f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-thswx\" (UID: \"854f323b-efd5-4bac-8ffa-72bd000cfb8f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx"
Oct 30 19:36:10.553602 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.553575    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xv8m\" (UniqueName: \"kubernetes.io/projected/854f323b-efd5-4bac-8ffa-72bd000cfb8f-kube-api-access-6xv8m\") pod \"llmisvc-controller-manager-869d5dcc5d-thswx\" (UID: \"854f323b-efd5-4bac-8ffa-72bd000cfb8f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx"
Oct 30 19:36:10.633310 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.633247    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx"
Oct 30 19:36:10.661512 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.661486    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"]
Oct 30 19:36:10.664971 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.664951    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.687669 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.687642    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"]
Oct 30 19:36:10.742002 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.741966    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/61adfc21-f356-4abd-bd63-5f690ec2e708-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.742176 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.742110    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4pdt\" (UniqueName: \"kubernetes.io/projected/61adfc21-f356-4abd-bd63-5f690ec2e708-kube-api-access-r4pdt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.742244 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.742214    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/61adfc21-f356-4abd-bd63-5f690ec2e708-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.742290 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.742265    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/61adfc21-f356-4abd-bd63-5f690ec2e708-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.742342 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.742292    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61adfc21-f356-4abd-bd63-5f690ec2e708-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.742342 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.742309    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/61adfc21-f356-4abd-bd63-5f690ec2e708-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.742342 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.742328    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61adfc21-f356-4abd-bd63-5f690ec2e708-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.843339 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.843296    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/61adfc21-f356-4abd-bd63-5f690ec2e708-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.843339 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.843340    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/61adfc21-f356-4abd-bd63-5f690ec2e708-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.843587 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.843362    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61adfc21-f356-4abd-bd63-5f690ec2e708-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.843587 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.843407    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/61adfc21-f356-4abd-bd63-5f690ec2e708-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.843587 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.843448    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61adfc21-f356-4abd-bd63-5f690ec2e708-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.843587 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.843482    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/61adfc21-f356-4abd-bd63-5f690ec2e708-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.843587 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.843549    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4pdt\" (UniqueName: \"kubernetes.io/projected/61adfc21-f356-4abd-bd63-5f690ec2e708-kube-api-access-r4pdt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.844193 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.844156    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/61adfc21-f356-4abd-bd63-5f690ec2e708-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.845672 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.845645    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/61adfc21-f356-4abd-bd63-5f690ec2e708-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.845871 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.845847    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61adfc21-f356-4abd-bd63-5f690ec2e708-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.845935 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.845883    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/61adfc21-f356-4abd-bd63-5f690ec2e708-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.845935 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.845896    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/61adfc21-f356-4abd-bd63-5f690ec2e708-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.856001 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.855981    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4pdt\" (UniqueName: \"kubernetes.io/projected/61adfc21-f356-4abd-bd63-5f690ec2e708-kube-api-access-r4pdt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.856114 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.855998    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61adfc21-f356-4abd-bd63-5f690ec2e708-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8w9xg\" (UID: \"61adfc21-f356-4abd-bd63-5f690ec2e708\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:10.907054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.906988    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx"]
Oct 30 19:36:10.910406 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:36:10.910383    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod854f323b_efd5_4bac_8ffa_72bd000cfb8f.slice/crio-8bae19de327e0e5b1610c3cebc9d524a554cb96c0c2cf85c31a5017a24df5ecc WatchSource:0}: Error finding container 8bae19de327e0e5b1610c3cebc9d524a554cb96c0c2cf85c31a5017a24df5ecc: Status 404 returned error can't find the container with id 8bae19de327e0e5b1610c3cebc9d524a554cb96c0c2cf85c31a5017a24df5ecc
Oct 30 19:36:10.911660 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.911642    5408 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:36:10.976863 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:10.976838    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:11.258920 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:11.258891    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"]
Oct 30 19:36:11.263238 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:36:11.263213    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61adfc21_f356_4abd_bd63_5f690ec2e708.slice/crio-4afddabb2a6a9374e21397d2a261efc9e4f0fe0d167d8445f9bb5f46d880c7a5 WatchSource:0}: Error finding container 4afddabb2a6a9374e21397d2a261efc9e4f0fe0d167d8445f9bb5f46d880c7a5: Status 404 returned error can't find the container with id 4afddabb2a6a9374e21397d2a261efc9e4f0fe0d167d8445f9bb5f46d880c7a5
Oct 30 19:36:11.265345 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:11.265315    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:36:11.265416 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:11.265374    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:36:11.729263 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:11.729159    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg" event={"ID":"61adfc21-f356-4abd-bd63-5f690ec2e708","Type":"ContainerStarted","Data":"ee7025e99a286ffd0aaf169c3eab63e92ec4cdd4ed69ce106fd1070e57a1dfca"}
Oct 30 19:36:11.729263 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:11.729230    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg" event={"ID":"61adfc21-f356-4abd-bd63-5f690ec2e708","Type":"ContainerStarted","Data":"4afddabb2a6a9374e21397d2a261efc9e4f0fe0d167d8445f9bb5f46d880c7a5"}
Oct 30 19:36:11.729843 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:11.729331    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:11.731136 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:11.731109    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx" event={"ID":"854f323b-efd5-4bac-8ffa-72bd000cfb8f","Type":"ContainerStarted","Data":"8bae19de327e0e5b1610c3cebc9d524a554cb96c0c2cf85c31a5017a24df5ecc"}
Oct 30 19:36:11.764567 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:11.764498    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg" podStartSLOduration=1.7644820860000001 podStartE2EDuration="1.764482086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:11.7624813 +0000 UTC m=+326.346750918" watchObservedRunningTime="2026-10-30 19:36:11.764482086 +0000 UTC m=+326.348751705"
Oct 30 19:36:12.737186 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:12.737155    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8w9xg"
Oct 30 19:36:12.798247 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:12.798203    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"]
Oct 30 19:36:12.798476 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:12.798445    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq" podUID="46163f29-52b0-4ec1-a104-2ad8fc1d66b7" containerName="discovery" containerID="cri-o://36fdbe916d76651ea7824ee924ccfdc787a27e5796885dd13f67d6c729e7307c" gracePeriod=30
Oct 30 19:36:13.037745 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.037720    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:36:13.164912 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.164881    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-cacerts\" (UniqueName: \"kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-cacerts\") pod \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") "
Oct 30 19:36:13.164912 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.164919    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-csr-ca-configmap\") pod \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") "
Oct 30 19:36:13.165166 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.164948    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-csr-dns-cert\") pod \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") "
Oct 30 19:36:13.165166 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.164997    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-local-certs\") pod \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") "
Oct 30 19:36:13.165166 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.165019    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-kube-api-access-csdvd\" (UniqueName: \"kubernetes.io/projected/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-kube-api-access-csdvd\") pod \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") "
Oct 30 19:36:13.165166 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.165073    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-token\" (UniqueName: \"kubernetes.io/projected/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-token\") pod \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") "
Oct 30 19:36:13.165166 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.165109    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-kubeconfig\") pod \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\" (UID: \"46163f29-52b0-4ec1-a104-2ad8fc1d66b7\") "
Oct 30 19:36:13.165425 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.165394    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-csr-ca-configmap" pod "46163f29-52b0-4ec1-a104-2ad8fc1d66b7" (UID: "46163f29-52b0-4ec1-a104-2ad8fc1d66b7"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:13.167387 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.167346    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-local-certs" pod "46163f29-52b0-4ec1-a104-2ad8fc1d66b7" (UID: "46163f29-52b0-4ec1-a104-2ad8fc1d66b7"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:13.167508 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.167429    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-kube-api-access-csdvd" pod "46163f29-52b0-4ec1-a104-2ad8fc1d66b7" (UID: "46163f29-52b0-4ec1-a104-2ad8fc1d66b7"). InnerVolumeSpecName "kube-api-access-csdvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:13.167672 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.167634    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-csr-dns-cert" pod "46163f29-52b0-4ec1-a104-2ad8fc1d66b7" (UID: "46163f29-52b0-4ec1-a104-2ad8fc1d66b7"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:13.167672 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.167663    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-cacerts" pod "46163f29-52b0-4ec1-a104-2ad8fc1d66b7" (UID: "46163f29-52b0-4ec1-a104-2ad8fc1d66b7"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:13.167849 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.167700    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-token" pod "46163f29-52b0-4ec1-a104-2ad8fc1d66b7" (UID: "46163f29-52b0-4ec1-a104-2ad8fc1d66b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:13.167849 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.167729    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-kubeconfig" pod "46163f29-52b0-4ec1-a104-2ad8fc1d66b7" (UID: "46163f29-52b0-4ec1-a104-2ad8fc1d66b7"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:13.266233 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.266205    5408 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-local-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:13.266233 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.266233    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csdvd\" (UniqueName: \"kubernetes.io/projected/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-kube-api-access-csdvd\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:13.266411 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.266243    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:13.266411 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.266253    5408 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-kubeconfig\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:13.266411 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.266264    5408 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-cacerts\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:13.266411 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.266272    5408 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-csr-ca-configmap\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:13.266411 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.266281    5408 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/46163f29-52b0-4ec1-a104-2ad8fc1d66b7-istio-csr-dns-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:13.739423 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.739390    5408 generic.go:356] "Generic (PLEG): container finished" podID="46163f29-52b0-4ec1-a104-2ad8fc1d66b7" containerID="36fdbe916d76651ea7824ee924ccfdc787a27e5796885dd13f67d6c729e7307c" exitCode=0
Oct 30 19:36:13.739867 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.739438    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"
Oct 30 19:36:13.739867 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.739478    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq" event={"ID":"46163f29-52b0-4ec1-a104-2ad8fc1d66b7","Type":"ContainerDied","Data":"36fdbe916d76651ea7824ee924ccfdc787a27e5796885dd13f67d6c729e7307c"}
Oct 30 19:36:13.739867 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.739538    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq" event={"ID":"46163f29-52b0-4ec1-a104-2ad8fc1d66b7","Type":"ContainerDied","Data":"85268447a0c08e4550f90f1e911d6b959c14f6bc253f530be10e0414717c8dd5"}
Oct 30 19:36:13.739867 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.739566    5408 scope.go:122] "RemoveContainer" containerID="36fdbe916d76651ea7824ee924ccfdc787a27e5796885dd13f67d6c729e7307c"
Oct 30 19:36:13.741240 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.741214    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx" event={"ID":"854f323b-efd5-4bac-8ffa-72bd000cfb8f","Type":"ContainerStarted","Data":"b6da3a6b8c290b46056cb40a20be1005e724304bfb53dc5cbe59ef3ebaa54c0f"}
Oct 30 19:36:13.741363 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.741313    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx"
Oct 30 19:36:13.747786 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.747768    5408 scope.go:122] "RemoveContainer" containerID="36fdbe916d76651ea7824ee924ccfdc787a27e5796885dd13f67d6c729e7307c"
Oct 30 19:36:13.748059 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:36:13.748023    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36fdbe916d76651ea7824ee924ccfdc787a27e5796885dd13f67d6c729e7307c\": container with ID starting with 36fdbe916d76651ea7824ee924ccfdc787a27e5796885dd13f67d6c729e7307c not found: ID does not exist" containerID="36fdbe916d76651ea7824ee924ccfdc787a27e5796885dd13f67d6c729e7307c"
Oct 30 19:36:13.748108 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.748072    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36fdbe916d76651ea7824ee924ccfdc787a27e5796885dd13f67d6c729e7307c"} err="failed to get container status \"36fdbe916d76651ea7824ee924ccfdc787a27e5796885dd13f67d6c729e7307c\": rpc error: code = NotFound desc = could not find container \"36fdbe916d76651ea7824ee924ccfdc787a27e5796885dd13f67d6c729e7307c\": container with ID starting with 36fdbe916d76651ea7824ee924ccfdc787a27e5796885dd13f67d6c729e7307c not found: ID does not exist"
Oct 30 19:36:13.760227 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.760208    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"]
Oct 30 19:36:13.764869 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.764851    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5c2dq"]
Oct 30 19:36:13.783466 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:13.783419    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx" podStartSLOduration=1.991402204 podStartE2EDuration="3.783404747s" totalImagesPullingTime="1.792002543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:13.783106774 +0000 UTC m=+328.367376391" watchObservedRunningTime="2026-10-30 19:36:13.783404747 +0000 UTC m=+328.367674366"
Oct 30 19:36:15.528816 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:15.528778    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46163f29-52b0-4ec1-a104-2ad8fc1d66b7" path="/var/lib/kubelet/pods/46163f29-52b0-4ec1-a104-2ad8fc1d66b7/volumes"
Oct 30 19:36:39.658390 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:36:39.658356    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:36:44.748454 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:36:44.748421    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx"
Oct 30 19:37:09.658561 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:37:09.658520    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:37:39.658481 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:37:39.658435    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:38:09.659942 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:38:09.659873    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:38:39.659244 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:38:39.659165    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:39:09.658696 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:39:09.658658    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:39:39.658434 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:39:39.658388    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:40:09.659187 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:40:09.659112    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:40:39.658991 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:40:39.658956    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:41:09.658563 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:41:09.658525    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:41:39.658671 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:41:39.658584    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:42:09.659077 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:42:09.659024    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:42:39.658447 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:42:39.658414    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:43:09.658256 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:43:09.658174    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:43:39.661487 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:43:39.661438    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:44:09.658882 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:44:09.658840    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:44:39.658305 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:44:39.658271    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:45:09.658834 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:45:09.658794    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:45:39.658101 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:45:39.658063    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:45:50.295334 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.295298    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"]
Oct 30 19:45:50.295719 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.295580    5408 cpu_manager.go:414] "containerMap: removing container" podUID="46163f29-52b0-4ec1-a104-2ad8fc1d66b7" containerName="discovery"
Oct 30 19:45:50.295719 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.295590    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46163f29-52b0-4ec1-a104-2ad8fc1d66b7" containerName="discovery"
Oct 30 19:45:50.295719 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.295680    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="46163f29-52b0-4ec1-a104-2ad8fc1d66b7" containerName="discovery"
Oct 30 19:45:50.298511 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.298494    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"
Oct 30 19:45:50.308476 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.308430    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-q2knc\""
Oct 30 19:45:50.311815 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.311792    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"]
Oct 30 19:45:50.312995 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.312973    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 19:45:50.313124 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.313043    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 19:45:50.313124 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.313095    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 19:45:50.385799 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.385760    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cdpq\" (UniqueName: \"kubernetes.io/projected/7298b613-173b-4139-a18e-ab39e73b9dae-kube-api-access-6cdpq\") pod \"limitador-limitador-74dfcfb894-4jkbg\" (UID: \"7298b613-173b-4139-a18e-ab39e73b9dae\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"
Oct 30 19:45:50.385961 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.385818    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7298b613-173b-4139-a18e-ab39e73b9dae-config-file\") pod \"limitador-limitador-74dfcfb894-4jkbg\" (UID: \"7298b613-173b-4139-a18e-ab39e73b9dae\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"
Oct 30 19:45:50.402742 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.402712    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"]
Oct 30 19:45:50.486769 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.486732    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cdpq\" (UniqueName: \"kubernetes.io/projected/7298b613-173b-4139-a18e-ab39e73b9dae-kube-api-access-6cdpq\") pod \"limitador-limitador-74dfcfb894-4jkbg\" (UID: \"7298b613-173b-4139-a18e-ab39e73b9dae\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"
Oct 30 19:45:50.486937 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.486810    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7298b613-173b-4139-a18e-ab39e73b9dae-config-file\") pod \"limitador-limitador-74dfcfb894-4jkbg\" (UID: \"7298b613-173b-4139-a18e-ab39e73b9dae\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"
Oct 30 19:45:50.487427 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.487389    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7298b613-173b-4139-a18e-ab39e73b9dae-config-file\") pod \"limitador-limitador-74dfcfb894-4jkbg\" (UID: \"7298b613-173b-4139-a18e-ab39e73b9dae\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"
Oct 30 19:45:50.499642 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.499614    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cdpq\" (UniqueName: \"kubernetes.io/projected/7298b613-173b-4139-a18e-ab39e73b9dae-kube-api-access-6cdpq\") pod \"limitador-limitador-74dfcfb894-4jkbg\" (UID: \"7298b613-173b-4139-a18e-ab39e73b9dae\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"
Oct 30 19:45:50.616196 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.616116    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"
Oct 30 19:45:50.894059 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.893966    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"]
Oct 30 19:45:50.898123 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:45:50.898094    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7298b613_173b_4139_a18e_ab39e73b9dae.slice/crio-29a47dc563ffd7077e00f0d909abbcccf0b9c53441f583732ac8affe933e1488 WatchSource:0}: Error finding container 29a47dc563ffd7077e00f0d909abbcccf0b9c53441f583732ac8affe933e1488: Status 404 returned error can't find the container with id 29a47dc563ffd7077e00f0d909abbcccf0b9c53441f583732ac8affe933e1488
Oct 30 19:45:50.900135 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:50.900118    5408 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:45:51.158988 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:51.158913    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ngdgz"]
Oct 30 19:45:51.163137 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:51.163118    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ngdgz"
Oct 30 19:45:51.170485 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:51.170463    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cq4cm\""
Oct 30 19:45:51.171146 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:51.171125    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ngdgz"]
Oct 30 19:45:51.191941 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:51.191916    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78vhh\" (UniqueName: \"kubernetes.io/projected/0ccbcb70-9004-490d-8c20-9aeb68974314-kube-api-access-78vhh\") pod \"authorino-57f7f8659f-ngdgz\" (UID: \"0ccbcb70-9004-490d-8c20-9aeb68974314\") " pod="kuadrant-system/authorino-57f7f8659f-ngdgz"
Oct 30 19:45:51.292826 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:51.292788    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78vhh\" (UniqueName: \"kubernetes.io/projected/0ccbcb70-9004-490d-8c20-9aeb68974314-kube-api-access-78vhh\") pod \"authorino-57f7f8659f-ngdgz\" (UID: \"0ccbcb70-9004-490d-8c20-9aeb68974314\") " pod="kuadrant-system/authorino-57f7f8659f-ngdgz"
Oct 30 19:45:51.306621 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:51.306594    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78vhh\" (UniqueName: \"kubernetes.io/projected/0ccbcb70-9004-490d-8c20-9aeb68974314-kube-api-access-78vhh\") pod \"authorino-57f7f8659f-ngdgz\" (UID: \"0ccbcb70-9004-490d-8c20-9aeb68974314\") " pod="kuadrant-system/authorino-57f7f8659f-ngdgz"
Oct 30 19:45:51.476081 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:51.475980    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ngdgz"
Oct 30 19:45:51.747891 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:51.747861    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ngdgz"]
Oct 30 19:45:51.751020 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:45:51.750991    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ccbcb70_9004_490d_8c20_9aeb68974314.slice/crio-3cf2a05fd6e1a01a0828804a7db68c888ea112d90633a73276aa52265d526efd WatchSource:0}: Error finding container 3cf2a05fd6e1a01a0828804a7db68c888ea112d90633a73276aa52265d526efd: Status 404 returned error can't find the container with id 3cf2a05fd6e1a01a0828804a7db68c888ea112d90633a73276aa52265d526efd
Oct 30 19:45:51.906743 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:51.906705    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ngdgz" event={"ID":"0ccbcb70-9004-490d-8c20-9aeb68974314","Type":"ContainerStarted","Data":"3cf2a05fd6e1a01a0828804a7db68c888ea112d90633a73276aa52265d526efd"}
Oct 30 19:45:51.908186 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:51.908157    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4jkbg" event={"ID":"7298b613-173b-4139-a18e-ab39e73b9dae","Type":"ContainerStarted","Data":"29a47dc563ffd7077e00f0d909abbcccf0b9c53441f583732ac8affe933e1488"}
Oct 30 19:45:53.922764 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:53.922681    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4jkbg" event={"ID":"7298b613-173b-4139-a18e-ab39e73b9dae","Type":"ContainerStarted","Data":"98ec0eed66165ee0b11841f6ff292bb52d77855e68a2f26aaea36613ae526ba8"}
Oct 30 19:45:53.923263 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:53.922811    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"
Oct 30 19:45:54.637889 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:54.637836    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-4jkbg" podStartSLOduration=1.884922439 podStartE2EDuration="4.637818679s" totalImagesPullingTime="2.75289624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:53.945650664 +0000 UTC m=+908.529920283" watchObservedRunningTime="2026-10-30 19:45:54.637818679 +0000 UTC m=+909.222088318"
Oct 30 19:45:54.638192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:54.638173    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ngdgz"]
Oct 30 19:45:54.927823 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:54.927724    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ngdgz" event={"ID":"0ccbcb70-9004-490d-8c20-9aeb68974314","Type":"ContainerStarted","Data":"f96af67dc9ee7341c03f9263ef1814dab681c2fe7c16335ac8191639d150a400"}
Oct 30 19:45:54.945980 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:54.945927    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-ngdgz" podStartSLOduration=1.716277939 podStartE2EDuration="3.945911933s" totalImagesPullingTime="2.229633994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:54.945207601 +0000 UTC m=+909.529477217" watchObservedRunningTime="2026-10-30 19:45:54.945911933 +0000 UTC m=+909.530181540"
Oct 30 19:45:55.931421 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:55.931382    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-ngdgz" podUID="0ccbcb70-9004-490d-8c20-9aeb68974314" containerName="authorino" containerID="cri-o://f96af67dc9ee7341c03f9263ef1814dab681c2fe7c16335ac8191639d150a400" gracePeriod=30
Oct 30 19:45:56.168252 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:56.168225    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ngdgz"
Oct 30 19:45:56.228153 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:56.228061    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ccbcb70-9004-490d-8c20-9aeb68974314-kube-api-access-78vhh\" (UniqueName: \"kubernetes.io/projected/0ccbcb70-9004-490d-8c20-9aeb68974314-kube-api-access-78vhh\") pod \"0ccbcb70-9004-490d-8c20-9aeb68974314\" (UID: \"0ccbcb70-9004-490d-8c20-9aeb68974314\") "
Oct 30 19:45:56.230231 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:56.230208    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ccbcb70-9004-490d-8c20-9aeb68974314-kube-api-access-78vhh" pod "0ccbcb70-9004-490d-8c20-9aeb68974314" (UID: "0ccbcb70-9004-490d-8c20-9aeb68974314"). InnerVolumeSpecName "kube-api-access-78vhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:56.328718 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:56.328683    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78vhh\" (UniqueName: \"kubernetes.io/projected/0ccbcb70-9004-490d-8c20-9aeb68974314-kube-api-access-78vhh\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:56.935833 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:56.935750    5408 generic.go:356] "Generic (PLEG): container finished" podID="0ccbcb70-9004-490d-8c20-9aeb68974314" containerID="f96af67dc9ee7341c03f9263ef1814dab681c2fe7c16335ac8191639d150a400" exitCode=0
Oct 30 19:45:56.935833 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:56.935789    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ngdgz"
Oct 30 19:45:56.935833 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:56.935802    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ngdgz" event={"ID":"0ccbcb70-9004-490d-8c20-9aeb68974314","Type":"ContainerDied","Data":"f96af67dc9ee7341c03f9263ef1814dab681c2fe7c16335ac8191639d150a400"}
Oct 30 19:45:56.936329 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:56.935839    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ngdgz" event={"ID":"0ccbcb70-9004-490d-8c20-9aeb68974314","Type":"ContainerDied","Data":"3cf2a05fd6e1a01a0828804a7db68c888ea112d90633a73276aa52265d526efd"}
Oct 30 19:45:56.936329 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:56.935858    5408 scope.go:122] "RemoveContainer" containerID="f96af67dc9ee7341c03f9263ef1814dab681c2fe7c16335ac8191639d150a400"
Oct 30 19:45:56.944423 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:56.944407    5408 scope.go:122] "RemoveContainer" containerID="f96af67dc9ee7341c03f9263ef1814dab681c2fe7c16335ac8191639d150a400"
Oct 30 19:45:56.944687 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:45:56.944665    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f96af67dc9ee7341c03f9263ef1814dab681c2fe7c16335ac8191639d150a400\": container with ID starting with f96af67dc9ee7341c03f9263ef1814dab681c2fe7c16335ac8191639d150a400 not found: ID does not exist" containerID="f96af67dc9ee7341c03f9263ef1814dab681c2fe7c16335ac8191639d150a400"
Oct 30 19:45:56.944762 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:56.944700    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f96af67dc9ee7341c03f9263ef1814dab681c2fe7c16335ac8191639d150a400"} err="failed to get container status \"f96af67dc9ee7341c03f9263ef1814dab681c2fe7c16335ac8191639d150a400\": rpc error: code = NotFound desc = could not find container \"f96af67dc9ee7341c03f9263ef1814dab681c2fe7c16335ac8191639d150a400\": container with ID starting with f96af67dc9ee7341c03f9263ef1814dab681c2fe7c16335ac8191639d150a400 not found: ID does not exist"
Oct 30 19:45:56.958719 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:56.958694    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ngdgz"]
Oct 30 19:45:56.968529 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:56.968507    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ngdgz"]
Oct 30 19:45:57.529216 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:45:57.529185    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ccbcb70-9004-490d-8c20-9aeb68974314" path="/var/lib/kubelet/pods/0ccbcb70-9004-490d-8c20-9aeb68974314/volumes"
Oct 30 19:46:03.900857 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:03.900819    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"]
Oct 30 19:46:03.901349 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:03.901065    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-4jkbg" podUID="7298b613-173b-4139-a18e-ab39e73b9dae" containerName="limitador" containerID="cri-o://98ec0eed66165ee0b11841f6ff292bb52d77855e68a2f26aaea36613ae526ba8" gracePeriod=30
Oct 30 19:46:03.901806 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:03.901787    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"
Oct 30 19:46:04.436879 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:04.436855    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"
Oct 30 19:46:04.489714 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:04.489685    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7298b613-173b-4139-a18e-ab39e73b9dae-config-file\" (UniqueName: \"kubernetes.io/configmap/7298b613-173b-4139-a18e-ab39e73b9dae-config-file\") pod \"7298b613-173b-4139-a18e-ab39e73b9dae\" (UID: \"7298b613-173b-4139-a18e-ab39e73b9dae\") "
Oct 30 19:46:04.489868 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:04.489771    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7298b613-173b-4139-a18e-ab39e73b9dae-kube-api-access-6cdpq\" (UniqueName: \"kubernetes.io/projected/7298b613-173b-4139-a18e-ab39e73b9dae-kube-api-access-6cdpq\") pod \"7298b613-173b-4139-a18e-ab39e73b9dae\" (UID: \"7298b613-173b-4139-a18e-ab39e73b9dae\") "
Oct 30 19:46:04.490074 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:04.490019    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7298b613-173b-4139-a18e-ab39e73b9dae-config-file" pod "7298b613-173b-4139-a18e-ab39e73b9dae" (UID: "7298b613-173b-4139-a18e-ab39e73b9dae"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:04.491760 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:04.491733    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7298b613-173b-4139-a18e-ab39e73b9dae-kube-api-access-6cdpq" pod "7298b613-173b-4139-a18e-ab39e73b9dae" (UID: "7298b613-173b-4139-a18e-ab39e73b9dae"). InnerVolumeSpecName "kube-api-access-6cdpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:04.590748 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:04.590714    5408 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7298b613-173b-4139-a18e-ab39e73b9dae-config-file\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:04.590748 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:04.590742    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cdpq\" (UniqueName: \"kubernetes.io/projected/7298b613-173b-4139-a18e-ab39e73b9dae-kube-api-access-6cdpq\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:04.967592 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:04.967509    5408 generic.go:356] "Generic (PLEG): container finished" podID="7298b613-173b-4139-a18e-ab39e73b9dae" containerID="98ec0eed66165ee0b11841f6ff292bb52d77855e68a2f26aaea36613ae526ba8" exitCode=0
Oct 30 19:46:04.967592 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:04.967552    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"
Oct 30 19:46:04.967592 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:04.967567    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4jkbg" event={"ID":"7298b613-173b-4139-a18e-ab39e73b9dae","Type":"ContainerDied","Data":"98ec0eed66165ee0b11841f6ff292bb52d77855e68a2f26aaea36613ae526ba8"}
Oct 30 19:46:04.968096 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:04.967616    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4jkbg" event={"ID":"7298b613-173b-4139-a18e-ab39e73b9dae","Type":"ContainerDied","Data":"29a47dc563ffd7077e00f0d909abbcccf0b9c53441f583732ac8affe933e1488"}
Oct 30 19:46:04.968096 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:04.967640    5408 scope.go:122] "RemoveContainer" containerID="98ec0eed66165ee0b11841f6ff292bb52d77855e68a2f26aaea36613ae526ba8"
Oct 30 19:46:04.975582 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:04.975556    5408 scope.go:122] "RemoveContainer" containerID="98ec0eed66165ee0b11841f6ff292bb52d77855e68a2f26aaea36613ae526ba8"
Oct 30 19:46:04.975857 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:46:04.975833    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98ec0eed66165ee0b11841f6ff292bb52d77855e68a2f26aaea36613ae526ba8\": container with ID starting with 98ec0eed66165ee0b11841f6ff292bb52d77855e68a2f26aaea36613ae526ba8 not found: ID does not exist" containerID="98ec0eed66165ee0b11841f6ff292bb52d77855e68a2f26aaea36613ae526ba8"
Oct 30 19:46:04.975919 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:04.975869    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98ec0eed66165ee0b11841f6ff292bb52d77855e68a2f26aaea36613ae526ba8"} err="failed to get container status \"98ec0eed66165ee0b11841f6ff292bb52d77855e68a2f26aaea36613ae526ba8\": rpc error: code = NotFound desc = could not find container \"98ec0eed66165ee0b11841f6ff292bb52d77855e68a2f26aaea36613ae526ba8\": container with ID starting with 98ec0eed66165ee0b11841f6ff292bb52d77855e68a2f26aaea36613ae526ba8 not found: ID does not exist"
Oct 30 19:46:04.994175 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:04.994142    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"]
Oct 30 19:46:04.998566 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:04.998544    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4jkbg"]
Oct 30 19:46:05.529085 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:05.529053    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7298b613-173b-4139-a18e-ab39e73b9dae" path="/var/lib/kubelet/pods/7298b613-173b-4139-a18e-ab39e73b9dae/volumes"
Oct 30 19:46:07.133659 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.133623    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-z5bc2"]
Oct 30 19:46:07.134138 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.134050    5408 cpu_manager.go:414] "containerMap: removing container" podUID="0ccbcb70-9004-490d-8c20-9aeb68974314" containerName="authorino"
Oct 30 19:46:07.134138 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.134069    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ccbcb70-9004-490d-8c20-9aeb68974314" containerName="authorino"
Oct 30 19:46:07.134138 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.134104    5408 cpu_manager.go:414] "containerMap: removing container" podUID="7298b613-173b-4139-a18e-ab39e73b9dae" containerName="limitador"
Oct 30 19:46:07.134138 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.134111    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7298b613-173b-4139-a18e-ab39e73b9dae" containerName="limitador"
Oct 30 19:46:07.134359 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.134189    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ccbcb70-9004-490d-8c20-9aeb68974314" containerName="authorino"
Oct 30 19:46:07.134359 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.134206    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="7298b613-173b-4139-a18e-ab39e73b9dae" containerName="limitador"
Oct 30 19:46:07.138342 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.138318    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z5bc2"
Oct 30 19:46:07.150570 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.150547    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:46:07.151065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.151042    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:46:07.156244 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.156168    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-z5bc2"]
Oct 30 19:46:07.157968 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.157950    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 19:46:07.158089 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.158004    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-xh7xr\""
Oct 30 19:46:07.209380 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.209342    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgmcc\" (UniqueName: \"kubernetes.io/projected/50dfdefa-ac3c-4f7e-a440-d9669a993fb5-kube-api-access-vgmcc\") pod \"postgres-7cdbf6dc44-z5bc2\" (UID: \"50dfdefa-ac3c-4f7e-a440-d9669a993fb5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z5bc2"
Oct 30 19:46:07.209559 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.209461    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/50dfdefa-ac3c-4f7e-a440-d9669a993fb5-data\") pod \"postgres-7cdbf6dc44-z5bc2\" (UID: \"50dfdefa-ac3c-4f7e-a440-d9669a993fb5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z5bc2"
Oct 30 19:46:07.310459 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.310421    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgmcc\" (UniqueName: \"kubernetes.io/projected/50dfdefa-ac3c-4f7e-a440-d9669a993fb5-kube-api-access-vgmcc\") pod \"postgres-7cdbf6dc44-z5bc2\" (UID: \"50dfdefa-ac3c-4f7e-a440-d9669a993fb5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z5bc2"
Oct 30 19:46:07.310634 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.310485    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/50dfdefa-ac3c-4f7e-a440-d9669a993fb5-data\") pod \"postgres-7cdbf6dc44-z5bc2\" (UID: \"50dfdefa-ac3c-4f7e-a440-d9669a993fb5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z5bc2"
Oct 30 19:46:07.310895 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.310870    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/50dfdefa-ac3c-4f7e-a440-d9669a993fb5-data\") pod \"postgres-7cdbf6dc44-z5bc2\" (UID: \"50dfdefa-ac3c-4f7e-a440-d9669a993fb5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z5bc2"
Oct 30 19:46:07.324012 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.323974    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgmcc\" (UniqueName: \"kubernetes.io/projected/50dfdefa-ac3c-4f7e-a440-d9669a993fb5-kube-api-access-vgmcc\") pod \"postgres-7cdbf6dc44-z5bc2\" (UID: \"50dfdefa-ac3c-4f7e-a440-d9669a993fb5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z5bc2"
Oct 30 19:46:07.451931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.451836    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z5bc2"
Oct 30 19:46:07.732950 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.732867    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-z5bc2"]
Oct 30 19:46:07.736958 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:46:07.736928    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50dfdefa_ac3c_4f7e_a440_d9669a993fb5.slice/crio-8b8565ded5812438dab40c716bd8c58082e9b30e39c4c9b9fc7310a8e47cc583 WatchSource:0}: Error finding container 8b8565ded5812438dab40c716bd8c58082e9b30e39c4c9b9fc7310a8e47cc583: Status 404 returned error can't find the container with id 8b8565ded5812438dab40c716bd8c58082e9b30e39c4c9b9fc7310a8e47cc583
Oct 30 19:46:07.981647 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:07.981613    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z5bc2" event={"ID":"50dfdefa-ac3c-4f7e-a440-d9669a993fb5","Type":"ContainerStarted","Data":"8b8565ded5812438dab40c716bd8c58082e9b30e39c4c9b9fc7310a8e47cc583"}
Oct 30 19:46:09.662950 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:46:09.662884    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:46:13.007782 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:13.007749    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z5bc2" event={"ID":"50dfdefa-ac3c-4f7e-a440-d9669a993fb5","Type":"ContainerStarted","Data":"1536043d0dc9ea07b5b436078e336f9436506f35bcbdcee12258079ade5786c2"}
Oct 30 19:46:13.008334 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:13.007837    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z5bc2"
Oct 30 19:46:13.029003 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:13.028501    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z5bc2" podStartSLOduration=0.93934861 podStartE2EDuration="6.028487188s" totalImagesPullingTime="5.089138578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:13.027900745 +0000 UTC m=+927.612170363" watchObservedRunningTime="2026-10-30 19:46:13.028487188 +0000 UTC m=+927.612756844"
Oct 30 19:46:19.041876 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:19.041847    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z5bc2"
Oct 30 19:46:26.469857 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.469816    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-mtpmg"]
Oct 30 19:46:26.480155 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.480129    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-mtpmg"
Oct 30 19:46:26.486807 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.486788    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 19:46:26.487858 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.487840    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 19:46:26.487915 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.487852    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2fq9m\""
Oct 30 19:46:26.487915 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.487849    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 19:46:26.497289 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.497261    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-mtpmg"]
Oct 30 19:46:26.563958 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.563924    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/203d85d2-e9b4-4218-a625-205a9a8d6820-metrics-certs\") pod \"maas-controller-579cf7c6d-mtpmg\" (UID: \"203d85d2-e9b4-4218-a625-205a9a8d6820\") " pod="opendatahub/maas-controller-579cf7c6d-mtpmg"
Oct 30 19:46:26.564132 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.564053    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/203d85d2-e9b4-4218-a625-205a9a8d6820-webhook-cert\") pod \"maas-controller-579cf7c6d-mtpmg\" (UID: \"203d85d2-e9b4-4218-a625-205a9a8d6820\") " pod="opendatahub/maas-controller-579cf7c6d-mtpmg"
Oct 30 19:46:26.564132 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.564102    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5xtv\" (UniqueName: \"kubernetes.io/projected/203d85d2-e9b4-4218-a625-205a9a8d6820-kube-api-access-s5xtv\") pod \"maas-controller-579cf7c6d-mtpmg\" (UID: \"203d85d2-e9b4-4218-a625-205a9a8d6820\") " pod="opendatahub/maas-controller-579cf7c6d-mtpmg"
Oct 30 19:46:26.665304 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.665249    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/203d85d2-e9b4-4218-a625-205a9a8d6820-metrics-certs\") pod \"maas-controller-579cf7c6d-mtpmg\" (UID: \"203d85d2-e9b4-4218-a625-205a9a8d6820\") " pod="opendatahub/maas-controller-579cf7c6d-mtpmg"
Oct 30 19:46:26.665511 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.665327    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/203d85d2-e9b4-4218-a625-205a9a8d6820-webhook-cert\") pod \"maas-controller-579cf7c6d-mtpmg\" (UID: \"203d85d2-e9b4-4218-a625-205a9a8d6820\") " pod="opendatahub/maas-controller-579cf7c6d-mtpmg"
Oct 30 19:46:26.665511 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.665376    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xtv\" (UniqueName: \"kubernetes.io/projected/203d85d2-e9b4-4218-a625-205a9a8d6820-kube-api-access-s5xtv\") pod \"maas-controller-579cf7c6d-mtpmg\" (UID: \"203d85d2-e9b4-4218-a625-205a9a8d6820\") " pod="opendatahub/maas-controller-579cf7c6d-mtpmg"
Oct 30 19:46:26.667745 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.667716    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/203d85d2-e9b4-4218-a625-205a9a8d6820-metrics-certs\") pod \"maas-controller-579cf7c6d-mtpmg\" (UID: \"203d85d2-e9b4-4218-a625-205a9a8d6820\") " pod="opendatahub/maas-controller-579cf7c6d-mtpmg"
Oct 30 19:46:26.667875 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.667804    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/203d85d2-e9b4-4218-a625-205a9a8d6820-webhook-cert\") pod \"maas-controller-579cf7c6d-mtpmg\" (UID: \"203d85d2-e9b4-4218-a625-205a9a8d6820\") " pod="opendatahub/maas-controller-579cf7c6d-mtpmg"
Oct 30 19:46:26.678252 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.678229    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5xtv\" (UniqueName: \"kubernetes.io/projected/203d85d2-e9b4-4218-a625-205a9a8d6820-kube-api-access-s5xtv\") pod \"maas-controller-579cf7c6d-mtpmg\" (UID: \"203d85d2-e9b4-4218-a625-205a9a8d6820\") " pod="opendatahub/maas-controller-579cf7c6d-mtpmg"
Oct 30 19:46:26.792906 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.792872    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-mtpmg"
Oct 30 19:46:26.813892 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.813862    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6df954cb67-w57gl"]
Oct 30 19:46:26.818342 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.818320    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6df954cb67-w57gl"
Oct 30 19:46:26.831224 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.831195    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6df954cb67-w57gl"]
Oct 30 19:46:26.867082 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.867021    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e480f59-6b71-4efe-905f-b1959a4a3891-metrics-certs\") pod \"maas-controller-6df954cb67-w57gl\" (UID: \"2e480f59-6b71-4efe-905f-b1959a4a3891\") " pod="opendatahub/maas-controller-6df954cb67-w57gl"
Oct 30 19:46:26.867241 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.867113    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvp7s\" (UniqueName: \"kubernetes.io/projected/2e480f59-6b71-4efe-905f-b1959a4a3891-kube-api-access-bvp7s\") pod \"maas-controller-6df954cb67-w57gl\" (UID: \"2e480f59-6b71-4efe-905f-b1959a4a3891\") " pod="opendatahub/maas-controller-6df954cb67-w57gl"
Oct 30 19:46:26.867241 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.867173    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e480f59-6b71-4efe-905f-b1959a4a3891-webhook-cert\") pod \"maas-controller-6df954cb67-w57gl\" (UID: \"2e480f59-6b71-4efe-905f-b1959a4a3891\") " pod="opendatahub/maas-controller-6df954cb67-w57gl"
Oct 30 19:46:26.968487 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.968454    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e480f59-6b71-4efe-905f-b1959a4a3891-webhook-cert\") pod \"maas-controller-6df954cb67-w57gl\" (UID: \"2e480f59-6b71-4efe-905f-b1959a4a3891\") " pod="opendatahub/maas-controller-6df954cb67-w57gl"
Oct 30 19:46:26.968626 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.968512    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e480f59-6b71-4efe-905f-b1959a4a3891-metrics-certs\") pod \"maas-controller-6df954cb67-w57gl\" (UID: \"2e480f59-6b71-4efe-905f-b1959a4a3891\") " pod="opendatahub/maas-controller-6df954cb67-w57gl"
Oct 30 19:46:26.968626 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.968536    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvp7s\" (UniqueName: \"kubernetes.io/projected/2e480f59-6b71-4efe-905f-b1959a4a3891-kube-api-access-bvp7s\") pod \"maas-controller-6df954cb67-w57gl\" (UID: \"2e480f59-6b71-4efe-905f-b1959a4a3891\") " pod="opendatahub/maas-controller-6df954cb67-w57gl"
Oct 30 19:46:26.971011 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.970983    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e480f59-6b71-4efe-905f-b1959a4a3891-webhook-cert\") pod \"maas-controller-6df954cb67-w57gl\" (UID: \"2e480f59-6b71-4efe-905f-b1959a4a3891\") " pod="opendatahub/maas-controller-6df954cb67-w57gl"
Oct 30 19:46:26.971011 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.971006    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e480f59-6b71-4efe-905f-b1959a4a3891-metrics-certs\") pod \"maas-controller-6df954cb67-w57gl\" (UID: \"2e480f59-6b71-4efe-905f-b1959a4a3891\") " pod="opendatahub/maas-controller-6df954cb67-w57gl"
Oct 30 19:46:26.981287 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:26.981267    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvp7s\" (UniqueName: \"kubernetes.io/projected/2e480f59-6b71-4efe-905f-b1959a4a3891-kube-api-access-bvp7s\") pod \"maas-controller-6df954cb67-w57gl\" (UID: \"2e480f59-6b71-4efe-905f-b1959a4a3891\") " pod="opendatahub/maas-controller-6df954cb67-w57gl"
Oct 30 19:46:27.075214 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:27.075146    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-mtpmg"]
Oct 30 19:46:27.078269 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:46:27.078242    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203d85d2_e9b4_4218_a625_205a9a8d6820.slice/crio-11367089cf086d29dfe3c05e40eeb13e20f17186dd25532ea97ae9677a3c9003 WatchSource:0}: Error finding container 11367089cf086d29dfe3c05e40eeb13e20f17186dd25532ea97ae9677a3c9003: Status 404 returned error can't find the container with id 11367089cf086d29dfe3c05e40eeb13e20f17186dd25532ea97ae9677a3c9003
Oct 30 19:46:27.135274 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:27.135239    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6df954cb67-w57gl"
Oct 30 19:46:27.413257 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:27.413175    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6df954cb67-w57gl"]
Oct 30 19:46:27.416038 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:46:27.416000    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e480f59_6b71_4efe_905f_b1959a4a3891.slice/crio-4ed654cb2e73897bdacdd4bfe27d146a4085d47ff167af8acee8993faeb6f4f5 WatchSource:0}: Error finding container 4ed654cb2e73897bdacdd4bfe27d146a4085d47ff167af8acee8993faeb6f4f5: Status 404 returned error can't find the container with id 4ed654cb2e73897bdacdd4bfe27d146a4085d47ff167af8acee8993faeb6f4f5
Oct 30 19:46:28.070259 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:28.070167    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6df954cb67-w57gl" event={"ID":"2e480f59-6b71-4efe-905f-b1959a4a3891","Type":"ContainerStarted","Data":"4ed654cb2e73897bdacdd4bfe27d146a4085d47ff167af8acee8993faeb6f4f5"}
Oct 30 19:46:28.073585 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:28.073531    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-mtpmg" event={"ID":"203d85d2-e9b4-4218-a625-205a9a8d6820","Type":"ContainerStarted","Data":"11367089cf086d29dfe3c05e40eeb13e20f17186dd25532ea97ae9677a3c9003"}
Oct 30 19:46:30.084073 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:30.084018    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-mtpmg" event={"ID":"203d85d2-e9b4-4218-a625-205a9a8d6820","Type":"ContainerStarted","Data":"b46b1ec15a136be8825d64831c573c342ed2097a8b83794ea9125c259289a6ff"}
Oct 30 19:46:30.084383 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:30.084144    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-mtpmg"
Oct 30 19:46:30.108875 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:30.108808    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-mtpmg" podStartSLOduration=1.449707787 podStartE2EDuration="4.10879138s" totalImagesPullingTime="2.659083593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:30.106141611 +0000 UTC m=+944.690411242" watchObservedRunningTime="2026-10-30 19:46:30.10879138 +0000 UTC m=+944.693060998"
Oct 30 19:46:31.089446 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.089409    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6df954cb67-w57gl" event={"ID":"2e480f59-6b71-4efe-905f-b1959a4a3891","Type":"ContainerStarted","Data":"14144c57a3c7c6fb5b19d7d8838f35fb6f4575bd453f2bf814b3a6fb059457da"}
Oct 30 19:46:31.715810 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.715752    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6df954cb67-w57gl" podStartSLOduration=3.076919133 podStartE2EDuration="5.715736738s" totalImagesPullingTime="2.638817605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:31.135846211 +0000 UTC m=+945.720115828" watchObservedRunningTime="2026-10-30 19:46:31.715736738 +0000 UTC m=+946.300006390"
Oct 30 19:46:31.716379 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.716362    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"]
Oct 30 19:46:31.719566 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.719551    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"
Oct 30 19:46:31.731005 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.730981    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 19:46:31.731348 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.731330    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-p8wwj\""
Oct 30 19:46:31.732497 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.732476    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 19:46:31.732696 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.732653    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 19:46:31.732818 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.732792    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"]
Oct 30 19:46:31.809489 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.809458    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d6bc6fc8-3673-4fcd-8125-fa63a3ce034a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-px62s\" (UID: \"d6bc6fc8-3673-4fcd-8125-fa63a3ce034a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"
Oct 30 19:46:31.809635 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.809516    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d6bc6fc8-3673-4fcd-8125-fa63a3ce034a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-px62s\" (UID: \"d6bc6fc8-3673-4fcd-8125-fa63a3ce034a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"
Oct 30 19:46:31.809635 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.809577    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d6bc6fc8-3673-4fcd-8125-fa63a3ce034a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-px62s\" (UID: \"d6bc6fc8-3673-4fcd-8125-fa63a3ce034a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"
Oct 30 19:46:31.809635 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.809599    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jshdz\" (UniqueName: \"kubernetes.io/projected/d6bc6fc8-3673-4fcd-8125-fa63a3ce034a-kube-api-access-jshdz\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-px62s\" (UID: \"d6bc6fc8-3673-4fcd-8125-fa63a3ce034a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"
Oct 30 19:46:31.910910 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.910879    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d6bc6fc8-3673-4fcd-8125-fa63a3ce034a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-px62s\" (UID: \"d6bc6fc8-3673-4fcd-8125-fa63a3ce034a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"
Oct 30 19:46:31.910910 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.910915    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d6bc6fc8-3673-4fcd-8125-fa63a3ce034a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-px62s\" (UID: \"d6bc6fc8-3673-4fcd-8125-fa63a3ce034a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"
Oct 30 19:46:31.911145 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.910934    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jshdz\" (UniqueName: \"kubernetes.io/projected/d6bc6fc8-3673-4fcd-8125-fa63a3ce034a-kube-api-access-jshdz\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-px62s\" (UID: \"d6bc6fc8-3673-4fcd-8125-fa63a3ce034a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"
Oct 30 19:46:31.911202 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.911146    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d6bc6fc8-3673-4fcd-8125-fa63a3ce034a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-px62s\" (UID: \"d6bc6fc8-3673-4fcd-8125-fa63a3ce034a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"
Oct 30 19:46:31.911734 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.911710    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d6bc6fc8-3673-4fcd-8125-fa63a3ce034a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-px62s\" (UID: \"d6bc6fc8-3673-4fcd-8125-fa63a3ce034a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"
Oct 30 19:46:31.911835 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.911782    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d6bc6fc8-3673-4fcd-8125-fa63a3ce034a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-px62s\" (UID: \"d6bc6fc8-3673-4fcd-8125-fa63a3ce034a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"
Oct 30 19:46:31.913369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.913345    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d6bc6fc8-3673-4fcd-8125-fa63a3ce034a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-px62s\" (UID: \"d6bc6fc8-3673-4fcd-8125-fa63a3ce034a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"
Oct 30 19:46:31.928448 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:31.928426    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jshdz\" (UniqueName: \"kubernetes.io/projected/d6bc6fc8-3673-4fcd-8125-fa63a3ce034a-kube-api-access-jshdz\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-px62s\" (UID: \"d6bc6fc8-3673-4fcd-8125-fa63a3ce034a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"
Oct 30 19:46:32.034966 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:32.034934    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"
Oct 30 19:46:32.096497 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:32.096467    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6df954cb67-w57gl"
Oct 30 19:46:32.311492 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:32.311405    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"]
Oct 30 19:46:32.314666 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:46:32.314629    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6bc6fc8_3673_4fcd_8125_fa63a3ce034a.slice/crio-01adee08bfd5b7b6e92118d7ebc68435043c926cf60722dd2752cad976637f95 WatchSource:0}: Error finding container 01adee08bfd5b7b6e92118d7ebc68435043c926cf60722dd2752cad976637f95: Status 404 returned error can't find the container with id 01adee08bfd5b7b6e92118d7ebc68435043c926cf60722dd2752cad976637f95
Oct 30 19:46:33.101310 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:33.101274    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s" event={"ID":"d6bc6fc8-3673-4fcd-8125-fa63a3ce034a","Type":"ContainerStarted","Data":"01adee08bfd5b7b6e92118d7ebc68435043c926cf60722dd2752cad976637f95"}
Oct 30 19:46:39.663938 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:46:39.663889    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:46:41.092653 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:41.092622    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-mtpmg"
Oct 30 19:46:43.103826 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:43.103795    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6df954cb67-w57gl"
Oct 30 19:46:43.160069 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:43.160013    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-mtpmg"]
Oct 30 19:46:43.160623 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:43.160581    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-mtpmg" podUID="203d85d2-e9b4-4218-a625-205a9a8d6820" containerName="manager" containerID="cri-o://b46b1ec15a136be8825d64831c573c342ed2097a8b83794ea9125c259289a6ff" gracePeriod=30
Oct 30 19:46:43.425707 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:43.425682    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-mtpmg"
Oct 30 19:46:43.524136 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:43.524099    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/203d85d2-e9b4-4218-a625-205a9a8d6820-webhook-cert\" (UniqueName: \"kubernetes.io/secret/203d85d2-e9b4-4218-a625-205a9a8d6820-webhook-cert\") pod \"203d85d2-e9b4-4218-a625-205a9a8d6820\" (UID: \"203d85d2-e9b4-4218-a625-205a9a8d6820\") "
Oct 30 19:46:43.524343 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:43.524164    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/203d85d2-e9b4-4218-a625-205a9a8d6820-metrics-certs\" (UniqueName: \"kubernetes.io/secret/203d85d2-e9b4-4218-a625-205a9a8d6820-metrics-certs\") pod \"203d85d2-e9b4-4218-a625-205a9a8d6820\" (UID: \"203d85d2-e9b4-4218-a625-205a9a8d6820\") "
Oct 30 19:46:43.524343 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:43.524198    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/203d85d2-e9b4-4218-a625-205a9a8d6820-kube-api-access-s5xtv\" (UniqueName: \"kubernetes.io/projected/203d85d2-e9b4-4218-a625-205a9a8d6820-kube-api-access-s5xtv\") pod \"203d85d2-e9b4-4218-a625-205a9a8d6820\" (UID: \"203d85d2-e9b4-4218-a625-205a9a8d6820\") "
Oct 30 19:46:43.526522 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:43.526494    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/203d85d2-e9b4-4218-a625-205a9a8d6820-webhook-cert" pod "203d85d2-e9b4-4218-a625-205a9a8d6820" (UID: "203d85d2-e9b4-4218-a625-205a9a8d6820"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:43.527355 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:43.527322    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/203d85d2-e9b4-4218-a625-205a9a8d6820-metrics-certs" pod "203d85d2-e9b4-4218-a625-205a9a8d6820" (UID: "203d85d2-e9b4-4218-a625-205a9a8d6820"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:43.529533 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:43.529504    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/203d85d2-e9b4-4218-a625-205a9a8d6820-kube-api-access-s5xtv" pod "203d85d2-e9b4-4218-a625-205a9a8d6820" (UID: "203d85d2-e9b4-4218-a625-205a9a8d6820"). InnerVolumeSpecName "kube-api-access-s5xtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:43.625077 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:43.625042    5408 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/203d85d2-e9b4-4218-a625-205a9a8d6820-webhook-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:43.625077 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:43.625071    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/203d85d2-e9b4-4218-a625-205a9a8d6820-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:43.625077 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:43.625081    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5xtv\" (UniqueName: \"kubernetes.io/projected/203d85d2-e9b4-4218-a625-205a9a8d6820-kube-api-access-s5xtv\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:44.156216 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:44.156178    5408 generic.go:356] "Generic (PLEG): container finished" podID="203d85d2-e9b4-4218-a625-205a9a8d6820" containerID="b46b1ec15a136be8825d64831c573c342ed2097a8b83794ea9125c259289a6ff" exitCode=0
Oct 30 19:46:44.156928 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:44.156226    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-mtpmg"
Oct 30 19:46:44.156928 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:44.156255    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-mtpmg" event={"ID":"203d85d2-e9b4-4218-a625-205a9a8d6820","Type":"ContainerDied","Data":"b46b1ec15a136be8825d64831c573c342ed2097a8b83794ea9125c259289a6ff"}
Oct 30 19:46:44.156928 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:44.156306    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-mtpmg" event={"ID":"203d85d2-e9b4-4218-a625-205a9a8d6820","Type":"ContainerDied","Data":"11367089cf086d29dfe3c05e40eeb13e20f17186dd25532ea97ae9677a3c9003"}
Oct 30 19:46:44.156928 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:44.156331    5408 scope.go:122] "RemoveContainer" containerID="b46b1ec15a136be8825d64831c573c342ed2097a8b83794ea9125c259289a6ff"
Oct 30 19:46:44.158317 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:44.158291    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s" event={"ID":"d6bc6fc8-3673-4fcd-8125-fa63a3ce034a","Type":"ContainerStarted","Data":"9182a3ef3860f744c9779ec4f57df5dba28b3af9d02247804e2b264b91121055"}
Oct 30 19:46:44.158454 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:44.158437    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"
Oct 30 19:46:44.164723 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:44.164567    5408 scope.go:122] "RemoveContainer" containerID="b46b1ec15a136be8825d64831c573c342ed2097a8b83794ea9125c259289a6ff"
Oct 30 19:46:44.164886 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:46:44.164866    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b46b1ec15a136be8825d64831c573c342ed2097a8b83794ea9125c259289a6ff\": container with ID starting with b46b1ec15a136be8825d64831c573c342ed2097a8b83794ea9125c259289a6ff not found: ID does not exist" containerID="b46b1ec15a136be8825d64831c573c342ed2097a8b83794ea9125c259289a6ff"
Oct 30 19:46:44.164930 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:44.164894    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46b1ec15a136be8825d64831c573c342ed2097a8b83794ea9125c259289a6ff"} err="failed to get container status \"b46b1ec15a136be8825d64831c573c342ed2097a8b83794ea9125c259289a6ff\": rpc error: code = NotFound desc = could not find container \"b46b1ec15a136be8825d64831c573c342ed2097a8b83794ea9125c259289a6ff\": container with ID starting with b46b1ec15a136be8825d64831c573c342ed2097a8b83794ea9125c259289a6ff not found: ID does not exist"
Oct 30 19:46:44.176026 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:44.176004    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-mtpmg"]
Oct 30 19:46:44.182414 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:44.182394    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-mtpmg"]
Oct 30 19:46:44.201563 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:44.201519    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s" podStartSLOduration=1.832165094 podStartE2EDuration="13.201505421s" totalImagesPullingTime="11.369340327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:44.200756331 +0000 UTC m=+958.785025947" watchObservedRunningTime="2026-10-30 19:46:44.201505421 +0000 UTC m=+958.785775041"
Oct 30 19:46:45.530821 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:45.530787    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="203d85d2-e9b4-4218-a625-205a9a8d6820" path="/var/lib/kubelet/pods/203d85d2-e9b4-4218-a625-205a9a8d6820/volumes"
Oct 30 19:46:55.169586 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:46:55.169554    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-px62s"
Oct 30 19:47:05.158223 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.158187    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"]
Oct 30 19:47:05.158596 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.158514    5408 cpu_manager.go:414] "containerMap: removing container" podUID="203d85d2-e9b4-4218-a625-205a9a8d6820" containerName="manager"
Oct 30 19:47:05.158596 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.158525    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="203d85d2-e9b4-4218-a625-205a9a8d6820" containerName="manager"
Oct 30 19:47:05.158596 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.158579    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="203d85d2-e9b4-4218-a625-205a9a8d6820" containerName="manager"
Oct 30 19:47:05.161552 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.161533    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"
Oct 30 19:47:05.179946 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.179346    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:47:05.179946 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.179424    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 19:47:05.179946 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.179914    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47lpr\""
Oct 30 19:47:05.180771 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.180751    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"]
Oct 30 19:47:05.196051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.196017    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/164dfba5-c638-431b-814c-4e296fc1aa37-openshift-service-ca\") pod \"maas-api-6d669c7fcd-zsn8x\" (UID: \"164dfba5-c638-431b-814c-4e296fc1aa37\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"
Oct 30 19:47:05.196171 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.196108    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk4zw\" (UniqueName: \"kubernetes.io/projected/164dfba5-c638-431b-814c-4e296fc1aa37-kube-api-access-sk4zw\") pod \"maas-api-6d669c7fcd-zsn8x\" (UID: \"164dfba5-c638-431b-814c-4e296fc1aa37\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"
Oct 30 19:47:05.196171 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.196154    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/164dfba5-c638-431b-814c-4e296fc1aa37-maas-api-tls\") pod \"maas-api-6d669c7fcd-zsn8x\" (UID: \"164dfba5-c638-431b-814c-4e296fc1aa37\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"
Oct 30 19:47:05.196241 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.196178    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/164dfba5-c638-431b-814c-4e296fc1aa37-metrics-certs\") pod \"maas-api-6d669c7fcd-zsn8x\" (UID: \"164dfba5-c638-431b-814c-4e296fc1aa37\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"
Oct 30 19:47:05.297061 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.297012    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/164dfba5-c638-431b-814c-4e296fc1aa37-openshift-service-ca\") pod \"maas-api-6d669c7fcd-zsn8x\" (UID: \"164dfba5-c638-431b-814c-4e296fc1aa37\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"
Oct 30 19:47:05.297238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.297081    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk4zw\" (UniqueName: \"kubernetes.io/projected/164dfba5-c638-431b-814c-4e296fc1aa37-kube-api-access-sk4zw\") pod \"maas-api-6d669c7fcd-zsn8x\" (UID: \"164dfba5-c638-431b-814c-4e296fc1aa37\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"
Oct 30 19:47:05.297238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.297139    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/164dfba5-c638-431b-814c-4e296fc1aa37-maas-api-tls\") pod \"maas-api-6d669c7fcd-zsn8x\" (UID: \"164dfba5-c638-431b-814c-4e296fc1aa37\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"
Oct 30 19:47:05.297238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.297169    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/164dfba5-c638-431b-814c-4e296fc1aa37-metrics-certs\") pod \"maas-api-6d669c7fcd-zsn8x\" (UID: \"164dfba5-c638-431b-814c-4e296fc1aa37\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"
Oct 30 19:47:05.297771 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.297741    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/164dfba5-c638-431b-814c-4e296fc1aa37-openshift-service-ca\") pod \"maas-api-6d669c7fcd-zsn8x\" (UID: \"164dfba5-c638-431b-814c-4e296fc1aa37\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"
Oct 30 19:47:05.299587 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.299557    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/164dfba5-c638-431b-814c-4e296fc1aa37-maas-api-tls\") pod \"maas-api-6d669c7fcd-zsn8x\" (UID: \"164dfba5-c638-431b-814c-4e296fc1aa37\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"
Oct 30 19:47:05.299705 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.299651    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/164dfba5-c638-431b-814c-4e296fc1aa37-metrics-certs\") pod \"maas-api-6d669c7fcd-zsn8x\" (UID: \"164dfba5-c638-431b-814c-4e296fc1aa37\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"
Oct 30 19:47:05.309346 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.309321    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk4zw\" (UniqueName: \"kubernetes.io/projected/164dfba5-c638-431b-814c-4e296fc1aa37-kube-api-access-sk4zw\") pod \"maas-api-6d669c7fcd-zsn8x\" (UID: \"164dfba5-c638-431b-814c-4e296fc1aa37\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"
Oct 30 19:47:05.474007 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.473941    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"
Oct 30 19:47:05.762065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:05.762010    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"]
Oct 30 19:47:05.766350 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:47:05.766322    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164dfba5_c638_431b_814c_4e296fc1aa37.slice/crio-b040ba8b44118ac004b6633f3b0841c86424bea6828dc644ed43e245df9aeed2 WatchSource:0}: Error finding container b040ba8b44118ac004b6633f3b0841c86424bea6828dc644ed43e245df9aeed2: Status 404 returned error can't find the container with id b040ba8b44118ac004b6633f3b0841c86424bea6828dc644ed43e245df9aeed2
Oct 30 19:47:06.242789 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:06.242756    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x" event={"ID":"164dfba5-c638-431b-814c-4e296fc1aa37","Type":"ContainerStarted","Data":"b040ba8b44118ac004b6633f3b0841c86424bea6828dc644ed43e245df9aeed2"}
Oct 30 19:47:08.251453 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:08.251416    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x" event={"ID":"164dfba5-c638-431b-814c-4e296fc1aa37","Type":"ContainerStarted","Data":"68cf35683412bfbc0c6b735efe1a685b7c6e70c4a4db63545eca73344458209f"}
Oct 30 19:47:08.251846 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:08.251526    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"
Oct 30 19:47:08.277659 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:08.277606    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x" podStartSLOduration=1.781691175 podStartE2EDuration="3.277592226s" totalImagesPullingTime="1.495901051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:08.275346074 +0000 UTC m=+982.859615692" watchObservedRunningTime="2026-10-30 19:47:08.277592226 +0000 UTC m=+982.861861843"
Oct 30 19:47:09.658884 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:47:09.658844    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:47:14.261206 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:14.261179    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"
Oct 30 19:47:15.140747 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.140713    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w5dlh"]
Oct 30 19:47:15.297760 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.297729    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w5dlh"]
Oct 30 19:47:15.297760 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.297765    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w5dlh"]
Oct 30 19:47:15.298274 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.297838    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w5dlh"
Oct 30 19:47:15.335250 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.335221    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w5dlh"
Oct 30 19:47:15.335479 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.335460    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"]
Oct 30 19:47:15.349403 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.349385    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"
Oct 30 19:47:15.363002 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.362980    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"]
Oct 30 19:47:15.382435 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.382411    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/057b7980-6217-4c2d-98b4-63e8deda405a-maas-api-tls\") pod \"maas-api-859f666b94-5kb4t\" (UID: \"057b7980-6217-4c2d-98b4-63e8deda405a\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"
Oct 30 19:47:15.382671 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.382638    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/057b7980-6217-4c2d-98b4-63e8deda405a-metrics-certs\") pod \"maas-api-859f666b94-5kb4t\" (UID: \"057b7980-6217-4c2d-98b4-63e8deda405a\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"
Oct 30 19:47:15.382769 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.382708    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/057b7980-6217-4c2d-98b4-63e8deda405a-openshift-service-ca\") pod \"maas-api-859f666b94-5kb4t\" (UID: \"057b7980-6217-4c2d-98b4-63e8deda405a\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"
Oct 30 19:47:15.382769 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.382746    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzjfr\" (UniqueName: \"kubernetes.io/projected/057b7980-6217-4c2d-98b4-63e8deda405a-kube-api-access-kzjfr\") pod \"maas-api-859f666b94-5kb4t\" (UID: \"057b7980-6217-4c2d-98b4-63e8deda405a\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"
Oct 30 19:47:15.483761 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.483686    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/057b7980-6217-4c2d-98b4-63e8deda405a-maas-api-tls\") pod \"maas-api-859f666b94-5kb4t\" (UID: \"057b7980-6217-4c2d-98b4-63e8deda405a\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"
Oct 30 19:47:15.483761 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.483732    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/057b7980-6217-4c2d-98b4-63e8deda405a-metrics-certs\") pod \"maas-api-859f666b94-5kb4t\" (UID: \"057b7980-6217-4c2d-98b4-63e8deda405a\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"
Oct 30 19:47:15.483761 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.483759    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/057b7980-6217-4c2d-98b4-63e8deda405a-openshift-service-ca\") pod \"maas-api-859f666b94-5kb4t\" (UID: \"057b7980-6217-4c2d-98b4-63e8deda405a\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"
Oct 30 19:47:15.483972 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.483775    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzjfr\" (UniqueName: \"kubernetes.io/projected/057b7980-6217-4c2d-98b4-63e8deda405a-kube-api-access-kzjfr\") pod \"maas-api-859f666b94-5kb4t\" (UID: \"057b7980-6217-4c2d-98b4-63e8deda405a\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"
Oct 30 19:47:15.484473 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.484446    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/057b7980-6217-4c2d-98b4-63e8deda405a-openshift-service-ca\") pod \"maas-api-859f666b94-5kb4t\" (UID: \"057b7980-6217-4c2d-98b4-63e8deda405a\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"
Oct 30 19:47:15.486139 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.486120    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/057b7980-6217-4c2d-98b4-63e8deda405a-metrics-certs\") pod \"maas-api-859f666b94-5kb4t\" (UID: \"057b7980-6217-4c2d-98b4-63e8deda405a\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"
Oct 30 19:47:15.486229 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.486166    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/057b7980-6217-4c2d-98b4-63e8deda405a-maas-api-tls\") pod \"maas-api-859f666b94-5kb4t\" (UID: \"057b7980-6217-4c2d-98b4-63e8deda405a\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"
Oct 30 19:47:15.503615 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.503595    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzjfr\" (UniqueName: \"kubernetes.io/projected/057b7980-6217-4c2d-98b4-63e8deda405a-kube-api-access-kzjfr\") pod \"maas-api-859f666b94-5kb4t\" (UID: \"057b7980-6217-4c2d-98b4-63e8deda405a\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"
Oct 30 19:47:15.561704 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.561681    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7545fb7886-j5fwg"]
Oct 30 19:47:15.585439 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.585419    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7545fb7886-j5fwg"]
Oct 30 19:47:15.585556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.585496    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7545fb7886-j5fwg"
Oct 30 19:47:15.592215 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.592192    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cq4cm\""
Oct 30 19:47:15.592509 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.592287    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 19:47:15.592509 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.592299    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 19:47:15.592509 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.592345    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 19:47:15.679889 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.679867    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"
Oct 30 19:47:15.685198 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.685180    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snmd7\" (UniqueName: \"kubernetes.io/projected/c9c19159-e05f-4755-b929-fd731f6aeaa6-kube-api-access-snmd7\") pod \"authorino-7545fb7886-j5fwg\" (UID: \"c9c19159-e05f-4755-b929-fd731f6aeaa6\") " pod="kuadrant-system/authorino-7545fb7886-j5fwg"
Oct 30 19:47:15.685266 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.685207    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c9c19159-e05f-4755-b929-fd731f6aeaa6-tls-cert\") pod \"authorino-7545fb7886-j5fwg\" (UID: \"c9c19159-e05f-4755-b929-fd731f6aeaa6\") " pod="kuadrant-system/authorino-7545fb7886-j5fwg"
Oct 30 19:47:15.786252 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.786224    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snmd7\" (UniqueName: \"kubernetes.io/projected/c9c19159-e05f-4755-b929-fd731f6aeaa6-kube-api-access-snmd7\") pod \"authorino-7545fb7886-j5fwg\" (UID: \"c9c19159-e05f-4755-b929-fd731f6aeaa6\") " pod="kuadrant-system/authorino-7545fb7886-j5fwg"
Oct 30 19:47:15.786391 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.786259    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c9c19159-e05f-4755-b929-fd731f6aeaa6-tls-cert\") pod \"authorino-7545fb7886-j5fwg\" (UID: \"c9c19159-e05f-4755-b929-fd731f6aeaa6\") " pod="kuadrant-system/authorino-7545fb7886-j5fwg"
Oct 30 19:47:15.788519 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.788501    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c9c19159-e05f-4755-b929-fd731f6aeaa6-tls-cert\") pod \"authorino-7545fb7886-j5fwg\" (UID: \"c9c19159-e05f-4755-b929-fd731f6aeaa6\") " pod="kuadrant-system/authorino-7545fb7886-j5fwg"
Oct 30 19:47:15.799456 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.799439    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snmd7\" (UniqueName: \"kubernetes.io/projected/c9c19159-e05f-4755-b929-fd731f6aeaa6-kube-api-access-snmd7\") pod \"authorino-7545fb7886-j5fwg\" (UID: \"c9c19159-e05f-4755-b929-fd731f6aeaa6\") " pod="kuadrant-system/authorino-7545fb7886-j5fwg"
Oct 30 19:47:15.916576 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.916543    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7545fb7886-j5fwg"
Oct 30 19:47:15.977785 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:15.977744    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"]
Oct 30 19:47:15.981394 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:47:15.981365    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057b7980_6217_4c2d_98b4_63e8deda405a.slice/crio-605bd3422b82c07f6a1fba488345a2cd5a8866153946810b194f13ccf0fff8fa WatchSource:0}: Error finding container 605bd3422b82c07f6a1fba488345a2cd5a8866153946810b194f13ccf0fff8fa: Status 404 returned error can't find the container with id 605bd3422b82c07f6a1fba488345a2cd5a8866153946810b194f13ccf0fff8fa
Oct 30 19:47:16.219747 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:16.219716    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7545fb7886-j5fwg"]
Oct 30 19:47:16.222662 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:47:16.222631    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9c19159_e05f_4755_b929_fd731f6aeaa6.slice/crio-cc512bffa1c79a5396089f8d9c8dc495c06faaf6744df915440a0ba9155e01ca WatchSource:0}: Error finding container cc512bffa1c79a5396089f8d9c8dc495c06faaf6744df915440a0ba9155e01ca: Status 404 returned error can't find the container with id cc512bffa1c79a5396089f8d9c8dc495c06faaf6744df915440a0ba9155e01ca
Oct 30 19:47:16.286634 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:16.286552    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7545fb7886-j5fwg" event={"ID":"c9c19159-e05f-4755-b929-fd731f6aeaa6","Type":"ContainerStarted","Data":"cc512bffa1c79a5396089f8d9c8dc495c06faaf6744df915440a0ba9155e01ca"}
Oct 30 19:47:16.287662 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:16.287637    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w5dlh"
Oct 30 19:47:16.287769 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:16.287685    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t" event={"ID":"057b7980-6217-4c2d-98b4-63e8deda405a","Type":"ContainerStarted","Data":"605bd3422b82c07f6a1fba488345a2cd5a8866153946810b194f13ccf0fff8fa"}
Oct 30 19:47:16.327437 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:16.327401    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w5dlh"]
Oct 30 19:47:16.331983 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:16.331961    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w5dlh"]
Oct 30 19:47:17.297601 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:17.297513    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7545fb7886-j5fwg" event={"ID":"c9c19159-e05f-4755-b929-fd731f6aeaa6","Type":"ContainerStarted","Data":"471dcee534a410230e9ebe100687bae008cca26616ee2de40dcd56fe766703b6"}
Oct 30 19:47:17.299193 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:17.299166    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t" event={"ID":"057b7980-6217-4c2d-98b4-63e8deda405a","Type":"ContainerStarted","Data":"368631c85c69b2235bdfd826f0c88513ea14c7143aec37ac53c53b2db03f56ca"}
Oct 30 19:47:17.299317 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:17.299219    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"
Oct 30 19:47:17.350725 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:17.350672    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7545fb7886-j5fwg" podStartSLOduration=1.5753357019999998 podStartE2EDuration="2.350658788s" totalImagesPullingTime="775.323086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:17.350145788 +0000 UTC m=+991.934415405" watchObservedRunningTime="2026-10-30 19:47:17.350658788 +0000 UTC m=+991.934928402"
Oct 30 19:47:17.417479 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:17.417427    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t" podStartSLOduration=1.840143256 podStartE2EDuration="2.417412296s" totalImagesPullingTime="577.26904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:17.414564403 +0000 UTC m=+991.998834020" watchObservedRunningTime="2026-10-30 19:47:17.417412296 +0000 UTC m=+992.001681910"
Oct 30 19:47:18.398628 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:18.398598    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-28gd4"]
Oct 30 19:47:18.464663 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:18.464632    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-28gd4"]
Oct 30 19:47:18.464808 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:18.464734    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-28gd4"
Oct 30 19:47:18.511138 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:18.511107    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57afffa1-1703-46a7-829e-241810a74b2a-cert\") pod \"llmisvc-controller-manager-6466c554d8-28gd4\" (UID: \"57afffa1-1703-46a7-829e-241810a74b2a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-28gd4"
Oct 30 19:47:18.511254 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:18.511149    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpgxj\" (UniqueName: \"kubernetes.io/projected/57afffa1-1703-46a7-829e-241810a74b2a-kube-api-access-kpgxj\") pod \"llmisvc-controller-manager-6466c554d8-28gd4\" (UID: \"57afffa1-1703-46a7-829e-241810a74b2a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-28gd4"
Oct 30 19:47:18.612548 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:18.612519    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57afffa1-1703-46a7-829e-241810a74b2a-cert\") pod \"llmisvc-controller-manager-6466c554d8-28gd4\" (UID: \"57afffa1-1703-46a7-829e-241810a74b2a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-28gd4"
Oct 30 19:47:18.612690 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:18.612555    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpgxj\" (UniqueName: \"kubernetes.io/projected/57afffa1-1703-46a7-829e-241810a74b2a-kube-api-access-kpgxj\") pod \"llmisvc-controller-manager-6466c554d8-28gd4\" (UID: \"57afffa1-1703-46a7-829e-241810a74b2a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-28gd4"
Oct 30 19:47:18.614930 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:18.614898    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57afffa1-1703-46a7-829e-241810a74b2a-cert\") pod \"llmisvc-controller-manager-6466c554d8-28gd4\" (UID: \"57afffa1-1703-46a7-829e-241810a74b2a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-28gd4"
Oct 30 19:47:18.626054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:18.626006    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpgxj\" (UniqueName: \"kubernetes.io/projected/57afffa1-1703-46a7-829e-241810a74b2a-kube-api-access-kpgxj\") pod \"llmisvc-controller-manager-6466c554d8-28gd4\" (UID: \"57afffa1-1703-46a7-829e-241810a74b2a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-28gd4"
Oct 30 19:47:18.786271 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:18.786241    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-28gd4"
Oct 30 19:47:19.073042 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:19.072957    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-28gd4"]
Oct 30 19:47:19.075842 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:47:19.075811    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57afffa1_1703_46a7_829e_241810a74b2a.slice/crio-bbafbd3c167be07813459342d956f9b30b734aed4b254f5c6cf55822cae7d69e WatchSource:0}: Error finding container bbafbd3c167be07813459342d956f9b30b734aed4b254f5c6cf55822cae7d69e: Status 404 returned error can't find the container with id bbafbd3c167be07813459342d956f9b30b734aed4b254f5c6cf55822cae7d69e
Oct 30 19:47:19.309542 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:19.309508    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-28gd4" event={"ID":"57afffa1-1703-46a7-829e-241810a74b2a","Type":"ContainerStarted","Data":"bbafbd3c167be07813459342d956f9b30b734aed4b254f5c6cf55822cae7d69e"}
Oct 30 19:47:20.314767 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:20.314729    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-28gd4" event={"ID":"57afffa1-1703-46a7-829e-241810a74b2a","Type":"ContainerStarted","Data":"aba84f9ee339385bf184628e0caaf0d1948361fbf925c24af7c0af79cedb818b"}
Oct 30 19:47:20.315227 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:20.314872    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-28gd4"
Oct 30 19:47:20.340495 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:20.340451    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-28gd4" podStartSLOduration=1.942616892 podStartE2EDuration="2.340437527s" totalImagesPullingTime="397.820635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:20.338313316 +0000 UTC m=+994.922582933" watchObservedRunningTime="2026-10-30 19:47:20.340437527 +0000 UTC m=+994.924707140"
Oct 30 19:47:23.311878 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:23.311848    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"
Oct 30 19:47:23.366476 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:23.366446    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"]
Oct 30 19:47:23.366661 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:23.366637    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x" podUID="164dfba5-c638-431b-814c-4e296fc1aa37" containerName="maas-api" containerID="cri-o://68cf35683412bfbc0c6b735efe1a685b7c6e70c4a4db63545eca73344458209f" gracePeriod=30
Oct 30 19:47:23.610842 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:23.610821    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"
Oct 30 19:47:23.751640 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:23.751611    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/164dfba5-c638-431b-814c-4e296fc1aa37-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/164dfba5-c638-431b-814c-4e296fc1aa37-maas-api-tls\") pod \"164dfba5-c638-431b-814c-4e296fc1aa37\" (UID: \"164dfba5-c638-431b-814c-4e296fc1aa37\") "
Oct 30 19:47:23.751833 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:23.751667    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/164dfba5-c638-431b-814c-4e296fc1aa37-kube-api-access-sk4zw\" (UniqueName: \"kubernetes.io/projected/164dfba5-c638-431b-814c-4e296fc1aa37-kube-api-access-sk4zw\") pod \"164dfba5-c638-431b-814c-4e296fc1aa37\" (UID: \"164dfba5-c638-431b-814c-4e296fc1aa37\") "
Oct 30 19:47:23.751833 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:23.751691    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/164dfba5-c638-431b-814c-4e296fc1aa37-metrics-certs\" (UniqueName: \"kubernetes.io/secret/164dfba5-c638-431b-814c-4e296fc1aa37-metrics-certs\") pod \"164dfba5-c638-431b-814c-4e296fc1aa37\" (UID: \"164dfba5-c638-431b-814c-4e296fc1aa37\") "
Oct 30 19:47:23.751833 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:23.751730    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/164dfba5-c638-431b-814c-4e296fc1aa37-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/164dfba5-c638-431b-814c-4e296fc1aa37-openshift-service-ca\") pod \"164dfba5-c638-431b-814c-4e296fc1aa37\" (UID: \"164dfba5-c638-431b-814c-4e296fc1aa37\") "
Oct 30 19:47:23.752166 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:23.752129    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164dfba5-c638-431b-814c-4e296fc1aa37-openshift-service-ca" pod "164dfba5-c638-431b-814c-4e296fc1aa37" (UID: "164dfba5-c638-431b-814c-4e296fc1aa37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:23.753915 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:23.753885    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/164dfba5-c638-431b-814c-4e296fc1aa37-metrics-certs" pod "164dfba5-c638-431b-814c-4e296fc1aa37" (UID: "164dfba5-c638-431b-814c-4e296fc1aa37"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:23.754011 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:23.753941    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/164dfba5-c638-431b-814c-4e296fc1aa37-maas-api-tls" pod "164dfba5-c638-431b-814c-4e296fc1aa37" (UID: "164dfba5-c638-431b-814c-4e296fc1aa37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:23.754011 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:23.753958    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164dfba5-c638-431b-814c-4e296fc1aa37-kube-api-access-sk4zw" pod "164dfba5-c638-431b-814c-4e296fc1aa37" (UID: "164dfba5-c638-431b-814c-4e296fc1aa37"). InnerVolumeSpecName "kube-api-access-sk4zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:23.853207 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:23.853144    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/164dfba5-c638-431b-814c-4e296fc1aa37-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:23.853207 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:23.853167    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk4zw\" (UniqueName: \"kubernetes.io/projected/164dfba5-c638-431b-814c-4e296fc1aa37-kube-api-access-sk4zw\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:23.853207 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:23.853177    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/164dfba5-c638-431b-814c-4e296fc1aa37-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:23.853207 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:23.853187    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/164dfba5-c638-431b-814c-4e296fc1aa37-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:24.332017 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:24.331979    5408 generic.go:356] "Generic (PLEG): container finished" podID="164dfba5-c638-431b-814c-4e296fc1aa37" containerID="68cf35683412bfbc0c6b735efe1a685b7c6e70c4a4db63545eca73344458209f" exitCode=0
Oct 30 19:47:24.332491 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:24.332023    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"
Oct 30 19:47:24.332491 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:24.332044    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x" event={"ID":"164dfba5-c638-431b-814c-4e296fc1aa37","Type":"ContainerDied","Data":"68cf35683412bfbc0c6b735efe1a685b7c6e70c4a4db63545eca73344458209f"}
Oct 30 19:47:24.332491 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:24.332118    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x" event={"ID":"164dfba5-c638-431b-814c-4e296fc1aa37","Type":"ContainerDied","Data":"b040ba8b44118ac004b6633f3b0841c86424bea6828dc644ed43e245df9aeed2"}
Oct 30 19:47:24.332491 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:24.332151    5408 scope.go:122] "RemoveContainer" containerID="68cf35683412bfbc0c6b735efe1a685b7c6e70c4a4db63545eca73344458209f"
Oct 30 19:47:24.341858 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:24.341838    5408 scope.go:122] "RemoveContainer" containerID="68cf35683412bfbc0c6b735efe1a685b7c6e70c4a4db63545eca73344458209f"
Oct 30 19:47:24.342177 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:47:24.342158    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68cf35683412bfbc0c6b735efe1a685b7c6e70c4a4db63545eca73344458209f\": container with ID starting with 68cf35683412bfbc0c6b735efe1a685b7c6e70c4a4db63545eca73344458209f not found: ID does not exist" containerID="68cf35683412bfbc0c6b735efe1a685b7c6e70c4a4db63545eca73344458209f"
Oct 30 19:47:24.342250 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:24.342185    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cf35683412bfbc0c6b735efe1a685b7c6e70c4a4db63545eca73344458209f"} err="failed to get container status \"68cf35683412bfbc0c6b735efe1a685b7c6e70c4a4db63545eca73344458209f\": rpc error: code = NotFound desc = could not find container \"68cf35683412bfbc0c6b735efe1a685b7c6e70c4a4db63545eca73344458209f\": container with ID starting with 68cf35683412bfbc0c6b735efe1a685b7c6e70c4a4db63545eca73344458209f not found: ID does not exist"
Oct 30 19:47:24.361946 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:24.361923    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"]
Oct 30 19:47:24.368009 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:24.367989    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-zsn8x"]
Oct 30 19:47:25.528799 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:25.528767    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="164dfba5-c638-431b-814c-4e296fc1aa37" path="/var/lib/kubelet/pods/164dfba5-c638-431b-814c-4e296fc1aa37/volumes"
Oct 30 19:47:39.658104 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:47:39.658014    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:47:41.059280 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.059239    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"]
Oct 30 19:47:41.059761 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.059717    5408 cpu_manager.go:414] "containerMap: removing container" podUID="164dfba5-c638-431b-814c-4e296fc1aa37" containerName="maas-api"
Oct 30 19:47:41.059761 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.059734    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="164dfba5-c638-431b-814c-4e296fc1aa37" containerName="maas-api"
Oct 30 19:47:41.059870 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.059854    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="164dfba5-c638-431b-814c-4e296fc1aa37" containerName="maas-api"
Oct 30 19:47:41.064079 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.064060    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.079121 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.079096    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"]
Oct 30 19:47:41.149174 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.149147    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fbh7r\""
Oct 30 19:47:41.194267 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.194229    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.194442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.194305    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adfef3ed-77c2-4817-9a2c-96403e22f83b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.194442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.194347    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.194442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.194368    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.194557 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.194497    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.194598 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.194584    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.194630 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.194615    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wxg7\" (UniqueName: \"kubernetes.io/projected/adfef3ed-77c2-4817-9a2c-96403e22f83b-kube-api-access-7wxg7\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.194668 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.194635    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.194668 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.194660    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.295317 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.295282    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.295476 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.295341    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.295476 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.295380    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wxg7\" (UniqueName: \"kubernetes.io/projected/adfef3ed-77c2-4817-9a2c-96403e22f83b-kube-api-access-7wxg7\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.295476 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.295407    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.295591 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.295480    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.295591 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.295517    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.295591 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.295546    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adfef3ed-77c2-4817-9a2c-96403e22f83b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.295724 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.295594    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.295724 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.295622    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.295921 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.295896    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.296018 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.295934    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.296111 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.296091    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.296171 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.296094    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.296368 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.296341    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adfef3ed-77c2-4817-9a2c-96403e22f83b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.297784 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.297760    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.298020 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.298004    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.309185 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.309160    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.309357 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.309317    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wxg7\" (UniqueName: \"kubernetes.io/projected/adfef3ed-77c2-4817-9a2c-96403e22f83b-kube-api-access-7wxg7\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2wd98\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.377890 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.377854    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:41.664293 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:41.664217    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"]
Oct 30 19:47:41.668132 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:47:41.668102    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfef3ed_77c2_4817_9a2c_96403e22f83b.slice/crio-65e490e58bddbc7cd4399ae29f5ba712db5b472df7bbb073ab9e4272c224dc0b WatchSource:0}: Error finding container 65e490e58bddbc7cd4399ae29f5ba712db5b472df7bbb073ab9e4272c224dc0b: Status 404 returned error can't find the container with id 65e490e58bddbc7cd4399ae29f5ba712db5b472df7bbb073ab9e4272c224dc0b
Oct 30 19:47:42.410963 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:42.410927    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98" event={"ID":"adfef3ed-77c2-4817-9a2c-96403e22f83b","Type":"ContainerStarted","Data":"65e490e58bddbc7cd4399ae29f5ba712db5b472df7bbb073ab9e4272c224dc0b"}
Oct 30 19:47:44.666645 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:44.666606    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:47:44.667063 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:44.666691    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:47:44.667063 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:44.666737    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:47:45.433215 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:45.433182    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98" event={"ID":"adfef3ed-77c2-4817-9a2c-96403e22f83b","Type":"ContainerStarted","Data":"b31d2af6fc155b5e0f2807f21faf641f3e6a09819c93ae77d88fa83ec9557df4"}
Oct 30 19:47:45.462778 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:45.462724    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98" podStartSLOduration=1.467100196 podStartE2EDuration="4.462695374s" totalImagesPullingTime="2.995595178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:45.46011539 +0000 UTC m=+1020.044385009" watchObservedRunningTime="2026-10-30 19:47:45.462695374 +0000 UTC m=+1020.046964995"
Oct 30 19:47:46.378250 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:46.378213    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:46.379849 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:46.379822    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-2wd98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 30 19:47:46.379959 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:46.379869    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98" podUID="adfef3ed-77c2-4817-9a2c-96403e22f83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 30 19:47:47.378306 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:47.378270    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-2wd98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 30 19:47:47.378682 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:47.378322    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98" podUID="adfef3ed-77c2-4817-9a2c-96403e22f83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 30 19:47:48.379220 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:48.379185    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-2wd98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 30 19:47:48.379583 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:48.379250    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98" podUID="adfef3ed-77c2-4817-9a2c-96403e22f83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 30 19:47:49.378545 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:49.378508    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-2wd98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 30 19:47:49.378725 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:49.378572    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98" podUID="adfef3ed-77c2-4817-9a2c-96403e22f83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 30 19:47:50.379097 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:50.379047    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-2wd98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 30 19:47:50.379500 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:50.379123    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98" podUID="adfef3ed-77c2-4817-9a2c-96403e22f83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 30 19:47:51.321584 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:51.321555    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-28gd4"
Oct 30 19:47:51.375143 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:51.375112    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx"]
Oct 30 19:47:51.375321 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:51.375301    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx" podUID="854f323b-efd5-4bac-8ffa-72bd000cfb8f" containerName="manager" containerID="cri-o://b6da3a6b8c290b46056cb40a20be1005e724304bfb53dc5cbe59ef3ebaa54c0f" gracePeriod=30
Oct 30 19:47:51.383280 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:51.383243    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:51.384231 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:51.384208    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:51.385293 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:51.385276    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 19:47:51.619248 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:51.619219    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx"
Oct 30 19:47:51.788378 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:51.788339    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854f323b-efd5-4bac-8ffa-72bd000cfb8f-kube-api-access-6xv8m\" (UniqueName: \"kubernetes.io/projected/854f323b-efd5-4bac-8ffa-72bd000cfb8f-kube-api-access-6xv8m\") pod \"854f323b-efd5-4bac-8ffa-72bd000cfb8f\" (UID: \"854f323b-efd5-4bac-8ffa-72bd000cfb8f\") "
Oct 30 19:47:51.788552 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:51.788430    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/854f323b-efd5-4bac-8ffa-72bd000cfb8f-cert\" (UniqueName: \"kubernetes.io/secret/854f323b-efd5-4bac-8ffa-72bd000cfb8f-cert\") pod \"854f323b-efd5-4bac-8ffa-72bd000cfb8f\" (UID: \"854f323b-efd5-4bac-8ffa-72bd000cfb8f\") "
Oct 30 19:47:51.790460 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:51.790430    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854f323b-efd5-4bac-8ffa-72bd000cfb8f-kube-api-access-6xv8m" pod "854f323b-efd5-4bac-8ffa-72bd000cfb8f" (UID: "854f323b-efd5-4bac-8ffa-72bd000cfb8f"). InnerVolumeSpecName "kube-api-access-6xv8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:51.790569 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:51.790479    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/854f323b-efd5-4bac-8ffa-72bd000cfb8f-cert" pod "854f323b-efd5-4bac-8ffa-72bd000cfb8f" (UID: "854f323b-efd5-4bac-8ffa-72bd000cfb8f"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:51.889109 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:51.889010    5408 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/854f323b-efd5-4bac-8ffa-72bd000cfb8f-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.889109 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:51.889055    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xv8m\" (UniqueName: \"kubernetes.io/projected/854f323b-efd5-4bac-8ffa-72bd000cfb8f-kube-api-access-6xv8m\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:52.464556 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:52.464520    5408 generic.go:356] "Generic (PLEG): container finished" podID="854f323b-efd5-4bac-8ffa-72bd000cfb8f" containerID="b6da3a6b8c290b46056cb40a20be1005e724304bfb53dc5cbe59ef3ebaa54c0f" exitCode=0
Oct 30 19:47:52.464975 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:52.464574    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx"
Oct 30 19:47:52.464975 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:52.464600    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx" event={"ID":"854f323b-efd5-4bac-8ffa-72bd000cfb8f","Type":"ContainerDied","Data":"b6da3a6b8c290b46056cb40a20be1005e724304bfb53dc5cbe59ef3ebaa54c0f"}
Oct 30 19:47:52.464975 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:52.464671    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx" event={"ID":"854f323b-efd5-4bac-8ffa-72bd000cfb8f","Type":"ContainerDied","Data":"8bae19de327e0e5b1610c3cebc9d524a554cb96c0c2cf85c31a5017a24df5ecc"}
Oct 30 19:47:52.464975 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:52.464694    5408 scope.go:122] "RemoveContainer" containerID="b6da3a6b8c290b46056cb40a20be1005e724304bfb53dc5cbe59ef3ebaa54c0f"
Oct 30 19:47:52.474061 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:52.474013    5408 scope.go:122] "RemoveContainer" containerID="b6da3a6b8c290b46056cb40a20be1005e724304bfb53dc5cbe59ef3ebaa54c0f"
Oct 30 19:47:52.474352 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:47:52.474333    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6da3a6b8c290b46056cb40a20be1005e724304bfb53dc5cbe59ef3ebaa54c0f\": container with ID starting with b6da3a6b8c290b46056cb40a20be1005e724304bfb53dc5cbe59ef3ebaa54c0f not found: ID does not exist" containerID="b6da3a6b8c290b46056cb40a20be1005e724304bfb53dc5cbe59ef3ebaa54c0f"
Oct 30 19:47:52.474416 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:52.474366    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6da3a6b8c290b46056cb40a20be1005e724304bfb53dc5cbe59ef3ebaa54c0f"} err="failed to get container status \"b6da3a6b8c290b46056cb40a20be1005e724304bfb53dc5cbe59ef3ebaa54c0f\": rpc error: code = NotFound desc = could not find container \"b6da3a6b8c290b46056cb40a20be1005e724304bfb53dc5cbe59ef3ebaa54c0f\": container with ID starting with b6da3a6b8c290b46056cb40a20be1005e724304bfb53dc5cbe59ef3ebaa54c0f not found: ID does not exist"
Oct 30 19:47:52.487192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:52.487165    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx"]
Oct 30 19:47:52.491869 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:52.491846    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-thswx"]
Oct 30 19:47:53.529098 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:53.529059    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="854f323b-efd5-4bac-8ffa-72bd000cfb8f" path="/var/lib/kubelet/pods/854f323b-efd5-4bac-8ffa-72bd000cfb8f/volumes"
Oct 30 19:47:56.111389 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.111358    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"]
Oct 30 19:47:56.111848 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.111829    5408 cpu_manager.go:414] "containerMap: removing container" podUID="854f323b-efd5-4bac-8ffa-72bd000cfb8f" containerName="manager"
Oct 30 19:47:56.111848 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.111844    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="854f323b-efd5-4bac-8ffa-72bd000cfb8f" containerName="manager"
Oct 30 19:47:56.111951 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.111920    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="854f323b-efd5-4bac-8ffa-72bd000cfb8f" containerName="manager"
Oct 30 19:47:56.116706 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.116676    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.122440 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.122409    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.122576 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.122452    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.122576 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.122480    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.122695 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.122579    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.122695 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.122626    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.122695 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.122670    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxhf9\" (UniqueName: \"kubernetes.io/projected/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-kube-api-access-mxhf9\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.122871 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.122696    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.122871 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.122724    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.122871 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.122752    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.212637 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.212604    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"]
Oct 30 19:47:56.223585 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.223558    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.223712 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.223593    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.223712 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.223632    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.223820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.223768    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.223932 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.223826    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.223932 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.223859    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.223932 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.223886    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.224103 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.223936    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxhf9\" (UniqueName: \"kubernetes.io/projected/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-kube-api-access-mxhf9\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.224103 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.223979    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.224103 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.224075    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.224496 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.224463    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.224496 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.224474    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.224633 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.224505    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.224906 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.224886    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.226460 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.226436    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.226589 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.226571    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.237070 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.237051    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.239471 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.239451    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxhf9\" (UniqueName: \"kubernetes.io/projected/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-kube-api-access-mxhf9\") pod \"maas-default-gateway-openshift-default-57f78bcd55-h68ls\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.429871 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.429783    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:56.711875 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.711792    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"]
Oct 30 19:47:56.715208 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:47:56.715184    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ba43a0_ddfc_459d_8e82_1c2e197557a0.slice/crio-57aba8b251da3d47d5e5d1bdd81f82eea5a4faf43ea6be22dd1367e7e6dec58f WatchSource:0}: Error finding container 57aba8b251da3d47d5e5d1bdd81f82eea5a4faf43ea6be22dd1367e7e6dec58f: Status 404 returned error can't find the container with id 57aba8b251da3d47d5e5d1bdd81f82eea5a4faf43ea6be22dd1367e7e6dec58f
Oct 30 19:47:56.717450 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.717413    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:47:56.717565 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.717503    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:47:56.717565 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:56.717547    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:47:57.489522 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:57.489483    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls" event={"ID":"e6ba43a0-ddfc-459d-8e82-1c2e197557a0","Type":"ContainerStarted","Data":"7b6a44087634fbb74888a6a62d5b26b76705c9b10cf5b2e0617fd8b0809bef8b"}
Oct 30 19:47:57.489907 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:57.489563    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls" event={"ID":"e6ba43a0-ddfc-459d-8e82-1c2e197557a0","Type":"ContainerStarted","Data":"57aba8b251da3d47d5e5d1bdd81f82eea5a4faf43ea6be22dd1367e7e6dec58f"}
Oct 30 19:47:57.530488 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:57.530428    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls" podStartSLOduration=1.53040968 podStartE2EDuration="1.53040968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:57.527610847 +0000 UTC m=+1032.111880465" watchObservedRunningTime="2026-10-30 19:47:57.53040968 +0000 UTC m=+1032.114679298"
Oct 30 19:47:58.430467 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:58.430422    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:47:58.431737 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:58.431715    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-h68ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:47:58.431793 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:58.431763    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls" podUID="e6ba43a0-ddfc-459d-8e82-1c2e197557a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 19:47:59.431111 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:59.431070    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-h68ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:47:59.431490 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:47:59.431129    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls" podUID="e6ba43a0-ddfc-459d-8e82-1c2e197557a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 19:48:00.430818 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:00.430778    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-h68ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:48:00.431059 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:00.430844    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls" podUID="e6ba43a0-ddfc-459d-8e82-1c2e197557a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 19:48:01.430680 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:01.430648    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-h68ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:48:01.431079 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:01.430701    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls" podUID="e6ba43a0-ddfc-459d-8e82-1c2e197557a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 19:48:02.431163 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:02.431127    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-h68ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:48:02.431558 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:02.431185    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls" podUID="e6ba43a0-ddfc-459d-8e82-1c2e197557a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 19:48:03.435010 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:03.434977    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:48:03.435413 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:03.435268    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:48:03.435890 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:03.435873    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 19:48:09.663506 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:48:09.663463    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:48:11.117176 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.117142    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"]
Oct 30 19:48:11.121274 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.120770    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.144542 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.144514    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"]
Oct 30 19:48:11.160498 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.160460    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.160644 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.160514    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.160644 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.160609    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.160644 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.160640    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.160826 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.160671    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf03122-99f0-4878-98ef-eaa9325402f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.160826 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.160699    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.160826 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.160746    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4l67\" (UniqueName: \"kubernetes.io/projected/6bf03122-99f0-4878-98ef-eaa9325402f3-kube-api-access-v4l67\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.160826 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.160765    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.160826 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.160787    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.261629 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.261598    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.261809 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.261644    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4l67\" (UniqueName: \"kubernetes.io/projected/6bf03122-99f0-4878-98ef-eaa9325402f3-kube-api-access-v4l67\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.261809 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.261672    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.261809 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.261699    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.261809 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.261759    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.261809 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.261782    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.261809 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.261805    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.262163 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.261826    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.262163 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.261865    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf03122-99f0-4878-98ef-eaa9325402f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.262255 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.262177    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.262255 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.262236    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.262394 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.262370    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.262529 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.262508    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.262672 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.262652    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf03122-99f0-4878-98ef-eaa9325402f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.264106 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.264088    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.264650 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.264621    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.274323 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.274294    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.274511 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.274493    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4l67\" (UniqueName: \"kubernetes.io/projected/6bf03122-99f0-4878-98ef-eaa9325402f3-kube-api-access-v4l67\") pod \"maas-default-gateway-openshift-default-57f78bcd55-mkl2p\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.435199 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.435116    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:11.718410 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.718342    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"]
Oct 30 19:48:11.721645 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:48:11.721620    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf03122_99f0_4878_98ef_eaa9325402f3.slice/crio-b305c5c572513b7932f88b84ea232569d31d213eb63ffdca8864787e664b71ae WatchSource:0}: Error finding container b305c5c572513b7932f88b84ea232569d31d213eb63ffdca8864787e664b71ae: Status 404 returned error can't find the container with id b305c5c572513b7932f88b84ea232569d31d213eb63ffdca8864787e664b71ae
Oct 30 19:48:11.723724 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.723693    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:48:11.723785 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.723760    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:48:11.723821 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:11.723788    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:48:12.581856 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:12.581819    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p" event={"ID":"6bf03122-99f0-4878-98ef-eaa9325402f3","Type":"ContainerStarted","Data":"bbb0acf128e447b9edce4e90fc12c091805c88cda80f919cefc3a610511f99af"}
Oct 30 19:48:12.582359 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:12.581876    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p" event={"ID":"6bf03122-99f0-4878-98ef-eaa9325402f3","Type":"ContainerStarted","Data":"b305c5c572513b7932f88b84ea232569d31d213eb63ffdca8864787e664b71ae"}
Oct 30 19:48:12.616847 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:12.616788    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p" podStartSLOduration=1.616768092 podStartE2EDuration="1.616768092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:12.615503442 +0000 UTC m=+1047.199773060" watchObservedRunningTime="2026-10-30 19:48:12.616768092 +0000 UTC m=+1047.201037713"
Oct 30 19:48:13.439057 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:13.437634    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:13.446052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:13.444928    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-mkl2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 19:48:13.446052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:13.444982    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p" podUID="6bf03122-99f0-4878-98ef-eaa9325402f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 19:48:14.436416 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:14.436383    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-mkl2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 19:48:14.436876 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:14.436440    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p" podUID="6bf03122-99f0-4878-98ef-eaa9325402f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 19:48:15.440052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:15.439402    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-mkl2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 19:48:15.440052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:15.439465    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p" podUID="6bf03122-99f0-4878-98ef-eaa9325402f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 19:48:16.435955 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:16.435922    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-mkl2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 19:48:16.438108 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:16.436220    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p" podUID="6bf03122-99f0-4878-98ef-eaa9325402f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 19:48:17.438055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:17.437312    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-mkl2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 19:48:17.438055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:17.437374    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p" podUID="6bf03122-99f0-4878-98ef-eaa9325402f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 19:48:19.435851 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:19.435799    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-mkl2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:48:19.436640 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:19.436594    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p" podUID="6bf03122-99f0-4878-98ef-eaa9325402f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:20.229054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:20.228314    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:20.229529 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:20.229505    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:20.232984 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:20.232963    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 19:48:37.686888 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:37.686851    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7545fb7886-j5fwg"]
Oct 30 19:48:37.687561 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:37.687077    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7545fb7886-j5fwg" podUID="c9c19159-e05f-4755-b929-fd731f6aeaa6" containerName="authorino" containerID="cri-o://471dcee534a410230e9ebe100687bae008cca26616ee2de40dcd56fe766703b6" gracePeriod=30
Oct 30 19:48:37.931025 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:37.930999    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7545fb7886-j5fwg"
Oct 30 19:48:37.999910 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:37.999881    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c19159-e05f-4755-b929-fd731f6aeaa6-kube-api-access-snmd7\" (UniqueName: \"kubernetes.io/projected/c9c19159-e05f-4755-b929-fd731f6aeaa6-kube-api-access-snmd7\") pod \"c9c19159-e05f-4755-b929-fd731f6aeaa6\" (UID: \"c9c19159-e05f-4755-b929-fd731f6aeaa6\") "
Oct 30 19:48:38.000084 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:37.999946    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c19159-e05f-4755-b929-fd731f6aeaa6-tls-cert\" (UniqueName: \"kubernetes.io/secret/c9c19159-e05f-4755-b929-fd731f6aeaa6-tls-cert\") pod \"c9c19159-e05f-4755-b929-fd731f6aeaa6\" (UID: \"c9c19159-e05f-4755-b929-fd731f6aeaa6\") "
Oct 30 19:48:38.002137 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:38.002107    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c19159-e05f-4755-b929-fd731f6aeaa6-kube-api-access-snmd7" pod "c9c19159-e05f-4755-b929-fd731f6aeaa6" (UID: "c9c19159-e05f-4755-b929-fd731f6aeaa6"). InnerVolumeSpecName "kube-api-access-snmd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:38.010446 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:38.010414    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c19159-e05f-4755-b929-fd731f6aeaa6-tls-cert" pod "c9c19159-e05f-4755-b929-fd731f6aeaa6" (UID: "c9c19159-e05f-4755-b929-fd731f6aeaa6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:38.101075 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:38.101009    5408 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c9c19159-e05f-4755-b929-fd731f6aeaa6-tls-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:38.101075 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:38.101067    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snmd7\" (UniqueName: \"kubernetes.io/projected/c9c19159-e05f-4755-b929-fd731f6aeaa6-kube-api-access-snmd7\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:38.754164 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:38.754130    5408 generic.go:356] "Generic (PLEG): container finished" podID="c9c19159-e05f-4755-b929-fd731f6aeaa6" containerID="471dcee534a410230e9ebe100687bae008cca26616ee2de40dcd56fe766703b6" exitCode=0
Oct 30 19:48:38.754525 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:38.754177    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7545fb7886-j5fwg" event={"ID":"c9c19159-e05f-4755-b929-fd731f6aeaa6","Type":"ContainerDied","Data":"471dcee534a410230e9ebe100687bae008cca26616ee2de40dcd56fe766703b6"}
Oct 30 19:48:38.754525 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:38.754185    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7545fb7886-j5fwg"
Oct 30 19:48:38.754525 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:38.754202    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7545fb7886-j5fwg" event={"ID":"c9c19159-e05f-4755-b929-fd731f6aeaa6","Type":"ContainerDied","Data":"cc512bffa1c79a5396089f8d9c8dc495c06faaf6744df915440a0ba9155e01ca"}
Oct 30 19:48:38.754525 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:38.754217    5408 scope.go:122] "RemoveContainer" containerID="471dcee534a410230e9ebe100687bae008cca26616ee2de40dcd56fe766703b6"
Oct 30 19:48:38.762703 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:38.762687    5408 scope.go:122] "RemoveContainer" containerID="471dcee534a410230e9ebe100687bae008cca26616ee2de40dcd56fe766703b6"
Oct 30 19:48:38.762938 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:48:38.762918    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471dcee534a410230e9ebe100687bae008cca26616ee2de40dcd56fe766703b6\": container with ID starting with 471dcee534a410230e9ebe100687bae008cca26616ee2de40dcd56fe766703b6 not found: ID does not exist" containerID="471dcee534a410230e9ebe100687bae008cca26616ee2de40dcd56fe766703b6"
Oct 30 19:48:38.762990 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:38.762946    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471dcee534a410230e9ebe100687bae008cca26616ee2de40dcd56fe766703b6"} err="failed to get container status \"471dcee534a410230e9ebe100687bae008cca26616ee2de40dcd56fe766703b6\": rpc error: code = NotFound desc = could not find container \"471dcee534a410230e9ebe100687bae008cca26616ee2de40dcd56fe766703b6\": container with ID starting with 471dcee534a410230e9ebe100687bae008cca26616ee2de40dcd56fe766703b6 not found: ID does not exist"
Oct 30 19:48:38.786635 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:38.786614    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7545fb7886-j5fwg"]
Oct 30 19:48:38.792962 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:38.792942    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7545fb7886-j5fwg"]
Oct 30 19:48:39.529368 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:48:39.529336    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c19159-e05f-4755-b929-fd731f6aeaa6" path="/var/lib/kubelet/pods/c9c19159-e05f-4755-b929-fd731f6aeaa6/volumes"
Oct 30 19:48:39.658010 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:48:39.657981    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:49:09.658487 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:49:09.658406    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:49:39.661360 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:49:39.661323    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:49:44.719202 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.719171    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"]
Oct 30 19:49:44.719727 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.719708    5408 cpu_manager.go:414] "containerMap: removing container" podUID="c9c19159-e05f-4755-b929-fd731f6aeaa6" containerName="authorino"
Oct 30 19:49:44.719815 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.719729    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c19159-e05f-4755-b929-fd731f6aeaa6" containerName="authorino"
Oct 30 19:49:44.719815 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.719808    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c19159-e05f-4755-b929-fd731f6aeaa6" containerName="authorino"
Oct 30 19:49:44.722998 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.722978    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"
Oct 30 19:49:44.731143 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.731125    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-d5e3e5-serving-cert\""
Oct 30 19:49:44.748758 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.748733    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"]
Oct 30 19:49:44.851469 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.851442    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c024192c-6410-4363-89b9-4b0e4fd5e4dc-maas-api-tls\") pod \"maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8\" (UID: \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"
Oct 30 19:49:44.851620 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.851479    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn42q\" (UniqueName: \"kubernetes.io/projected/c024192c-6410-4363-89b9-4b0e4fd5e4dc-kube-api-access-rn42q\") pod \"maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8\" (UID: \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"
Oct 30 19:49:44.851620 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.851502    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c024192c-6410-4363-89b9-4b0e4fd5e4dc-metrics-certs\") pod \"maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8\" (UID: \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"
Oct 30 19:49:44.851693 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.851667    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c024192c-6410-4363-89b9-4b0e4fd5e4dc-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8\" (UID: \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"
Oct 30 19:49:44.953491 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.953450    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c024192c-6410-4363-89b9-4b0e4fd5e4dc-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8\" (UID: \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"
Oct 30 19:49:44.953684 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.953544    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c024192c-6410-4363-89b9-4b0e4fd5e4dc-maas-api-tls\") pod \"maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8\" (UID: \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"
Oct 30 19:49:44.953684 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.953584    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn42q\" (UniqueName: \"kubernetes.io/projected/c024192c-6410-4363-89b9-4b0e4fd5e4dc-kube-api-access-rn42q\") pod \"maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8\" (UID: \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"
Oct 30 19:49:44.953684 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.953618    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c024192c-6410-4363-89b9-4b0e4fd5e4dc-metrics-certs\") pod \"maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8\" (UID: \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"
Oct 30 19:49:44.954704 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.954677    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c024192c-6410-4363-89b9-4b0e4fd5e4dc-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8\" (UID: \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"
Oct 30 19:49:44.960686 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.960646    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c024192c-6410-4363-89b9-4b0e4fd5e4dc-maas-api-tls\") pod \"maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8\" (UID: \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"
Oct 30 19:49:44.960811 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.960765    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c024192c-6410-4363-89b9-4b0e4fd5e4dc-metrics-certs\") pod \"maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8\" (UID: \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"
Oct 30 19:49:44.969445 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:44.969380    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn42q\" (UniqueName: \"kubernetes.io/projected/c024192c-6410-4363-89b9-4b0e4fd5e4dc-kube-api-access-rn42q\") pod \"maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8\" (UID: \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"
Oct 30 19:49:45.035606 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:45.035575    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"
Oct 30 19:49:45.325968 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:45.325937    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"]
Oct 30 19:49:45.329707 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:49:45.329679    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc024192c_6410_4363_89b9_4b0e4fd5e4dc.slice/crio-94d07dde771e5c1133f0a5adfad03c0833ff97e59683619ea177b12ac48e2c7e WatchSource:0}: Error finding container 94d07dde771e5c1133f0a5adfad03c0833ff97e59683619ea177b12ac48e2c7e: Status 404 returned error can't find the container with id 94d07dde771e5c1133f0a5adfad03c0833ff97e59683619ea177b12ac48e2c7e
Oct 30 19:49:46.043105 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:46.043072    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8" event={"ID":"c024192c-6410-4363-89b9-4b0e4fd5e4dc","Type":"ContainerStarted","Data":"674a72e698d47113300429da81abd44d46d9689797ce47f42c3c31d30d60ed0e"}
Oct 30 19:49:46.043594 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:46.043128    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8" event={"ID":"c024192c-6410-4363-89b9-4b0e4fd5e4dc","Type":"ContainerStarted","Data":"94d07dde771e5c1133f0a5adfad03c0833ff97e59683619ea177b12ac48e2c7e"}
Oct 30 19:49:46.043594 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:46.043201    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"
Oct 30 19:49:46.117962 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:46.117867    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8" podStartSLOduration=1.600707415 podStartE2EDuration="2.117851771s" totalImagesPullingTime="517.144356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:46.114562382 +0000 UTC m=+1140.698832010" watchObservedRunningTime="2026-10-30 19:49:46.117851771 +0000 UTC m=+1140.702121388"
Oct 30 19:49:46.841614 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:46.841581    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7"]
Oct 30 19:49:46.845124 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:46.845108    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7"
Oct 30 19:49:46.849569 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:46.849547    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ced832\""
Oct 30 19:49:46.862437 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:46.862396    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ced832-dockercfg-kxkvg\""
Oct 30 19:49:46.863765 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:46.863740    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7"]
Oct 30 19:49:46.983835 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:46.978545    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268a8f7e-737f-4806-9f80-7d7de4494b00-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7\" (UID: \"268a8f7e-737f-4806-9f80-7d7de4494b00\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7"
Oct 30 19:49:46.984141 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:46.984114    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzknb\" (UniqueName: \"kubernetes.io/projected/268a8f7e-737f-4806-9f80-7d7de4494b00-kube-api-access-mzknb\") pod \"payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7\" (UID: \"268a8f7e-737f-4806-9f80-7d7de4494b00\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7"
Oct 30 19:49:47.084974 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:47.084940    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268a8f7e-737f-4806-9f80-7d7de4494b00-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7\" (UID: \"268a8f7e-737f-4806-9f80-7d7de4494b00\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7"
Oct 30 19:49:47.084974 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:47.084980    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzknb\" (UniqueName: \"kubernetes.io/projected/268a8f7e-737f-4806-9f80-7d7de4494b00-kube-api-access-mzknb\") pod \"payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7\" (UID: \"268a8f7e-737f-4806-9f80-7d7de4494b00\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7"
Oct 30 19:49:47.085742 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:47.085720    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268a8f7e-737f-4806-9f80-7d7de4494b00-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7\" (UID: \"268a8f7e-737f-4806-9f80-7d7de4494b00\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7"
Oct 30 19:49:47.114308 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:47.114241    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzknb\" (UniqueName: \"kubernetes.io/projected/268a8f7e-737f-4806-9f80-7d7de4494b00-kube-api-access-mzknb\") pod \"payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7\" (UID: \"268a8f7e-737f-4806-9f80-7d7de4494b00\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7"
Oct 30 19:49:47.158328 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:47.158304    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7"
Oct 30 19:49:47.469696 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:47.469619    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7"]
Oct 30 19:49:47.473721 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:49:47.473697    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268a8f7e_737f_4806_9f80_7d7de4494b00.slice/crio-071ffee22aa020b5387416afab821fe4dc300e1747285149e7e6c9410471ffbe WatchSource:0}: Error finding container 071ffee22aa020b5387416afab821fe4dc300e1747285149e7e6c9410471ffbe: Status 404 returned error can't find the container with id 071ffee22aa020b5387416afab821fe4dc300e1747285149e7e6c9410471ffbe
Oct 30 19:49:48.054666 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.054617    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7" event={"ID":"268a8f7e-737f-4806-9f80-7d7de4494b00","Type":"ContainerStarted","Data":"071ffee22aa020b5387416afab821fe4dc300e1747285149e7e6c9410471ffbe"}
Oct 30 19:49:48.291336 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.291300    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"]
Oct 30 19:49:48.295022 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.294999    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"
Oct 30 19:49:48.300486 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.300459    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-3f57c5-serving-cert\""
Oct 30 19:49:48.335870 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.335797    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"]
Oct 30 19:49:48.398362 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.398330    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25efa968-ddb4-4893-8209-176519fb0224-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8\" (UID: \"25efa968-ddb4-4893-8209-176519fb0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"
Oct 30 19:49:48.398533 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.398397    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25efa968-ddb4-4893-8209-176519fb0224-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8\" (UID: \"25efa968-ddb4-4893-8209-176519fb0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"
Oct 30 19:49:48.398533 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.398505    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25efa968-ddb4-4893-8209-176519fb0224-metrics-certs\") pod \"maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8\" (UID: \"25efa968-ddb4-4893-8209-176519fb0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"
Oct 30 19:49:48.398649 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.398547    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vxqw\" (UniqueName: \"kubernetes.io/projected/25efa968-ddb4-4893-8209-176519fb0224-kube-api-access-2vxqw\") pod \"maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8\" (UID: \"25efa968-ddb4-4893-8209-176519fb0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"
Oct 30 19:49:48.502812 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.502780    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25efa968-ddb4-4893-8209-176519fb0224-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8\" (UID: \"25efa968-ddb4-4893-8209-176519fb0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"
Oct 30 19:49:48.503081 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.502829    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25efa968-ddb4-4893-8209-176519fb0224-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8\" (UID: \"25efa968-ddb4-4893-8209-176519fb0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"
Oct 30 19:49:48.503081 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.502876    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25efa968-ddb4-4893-8209-176519fb0224-metrics-certs\") pod \"maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8\" (UID: \"25efa968-ddb4-4893-8209-176519fb0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"
Oct 30 19:49:48.503081 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.502907    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vxqw\" (UniqueName: \"kubernetes.io/projected/25efa968-ddb4-4893-8209-176519fb0224-kube-api-access-2vxqw\") pod \"maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8\" (UID: \"25efa968-ddb4-4893-8209-176519fb0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"
Oct 30 19:49:48.504115 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.504084    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25efa968-ddb4-4893-8209-176519fb0224-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8\" (UID: \"25efa968-ddb4-4893-8209-176519fb0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"
Oct 30 19:49:48.507147 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.507120    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25efa968-ddb4-4893-8209-176519fb0224-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8\" (UID: \"25efa968-ddb4-4893-8209-176519fb0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"
Oct 30 19:49:48.507277 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.507222    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25efa968-ddb4-4893-8209-176519fb0224-metrics-certs\") pod \"maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8\" (UID: \"25efa968-ddb4-4893-8209-176519fb0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"
Oct 30 19:49:48.545048 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.544530    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vxqw\" (UniqueName: \"kubernetes.io/projected/25efa968-ddb4-4893-8209-176519fb0224-kube-api-access-2vxqw\") pod \"maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8\" (UID: \"25efa968-ddb4-4893-8209-176519fb0224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"
Oct 30 19:49:48.614881 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.614813    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"
Oct 30 19:49:48.967745 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:48.967662    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"]
Oct 30 19:49:48.995717 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:49:48.995685    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25efa968_ddb4_4893_8209_176519fb0224.slice/crio-1d0ea7fd5e14f9cb29c58dece16a42d5d8432e9758e5ee113338f5720ad660bf WatchSource:0}: Error finding container 1d0ea7fd5e14f9cb29c58dece16a42d5d8432e9758e5ee113338f5720ad660bf: Status 404 returned error can't find the container with id 1d0ea7fd5e14f9cb29c58dece16a42d5d8432e9758e5ee113338f5720ad660bf
Oct 30 19:49:49.061863 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:49.061826    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8" event={"ID":"25efa968-ddb4-4893-8209-176519fb0224","Type":"ContainerStarted","Data":"1d0ea7fd5e14f9cb29c58dece16a42d5d8432e9758e5ee113338f5720ad660bf"}
Oct 30 19:49:49.789304 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:49.789267    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"]
Oct 30 19:49:49.793463 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:49.793440    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"
Oct 30 19:49:49.810739 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:49.810712    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"]
Oct 30 19:49:49.811537 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:49.811244    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-8a8207-serving-cert\""
Oct 30 19:49:49.926569 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:49.925339    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx\" (UID: \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"
Oct 30 19:49:49.926569 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:49.925439    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcvsc\" (UniqueName: \"kubernetes.io/projected/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-kube-api-access-hcvsc\") pod \"maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx\" (UID: \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"
Oct 30 19:49:49.926569 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:49.925583    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-metrics-certs\") pod \"maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx\" (UID: \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"
Oct 30 19:49:49.926569 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:49.925626    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx\" (UID: \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"
Oct 30 19:49:50.026855 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.026822    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx\" (UID: \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"
Oct 30 19:49:50.026979 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.026943    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcvsc\" (UniqueName: \"kubernetes.io/projected/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-kube-api-access-hcvsc\") pod \"maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx\" (UID: \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"
Oct 30 19:49:50.027024 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.027007    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-metrics-certs\") pod \"maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx\" (UID: \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"
Oct 30 19:49:50.027087 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.027067    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx\" (UID: \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"
Oct 30 19:49:50.027706 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.027679    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx\" (UID: \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"
Oct 30 19:49:50.029072 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.029042    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx\" (UID: \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"
Oct 30 19:49:50.029172 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.029142    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-metrics-certs\") pod \"maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx\" (UID: \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"
Oct 30 19:49:50.039728 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.039684    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcvsc\" (UniqueName: \"kubernetes.io/projected/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-kube-api-access-hcvsc\") pod \"maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx\" (UID: \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"
Oct 30 19:49:50.067535 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.067509    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8" event={"ID":"25efa968-ddb4-4893-8209-176519fb0224","Type":"ContainerStarted","Data":"402d598b56ba3b38e165b41c30b8e8f9d111fde41e99a3784619b59bd57e9b2f"}
Oct 30 19:49:50.067649 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.067582    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"
Oct 30 19:49:50.068993 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.068970    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7" event={"ID":"268a8f7e-737f-4806-9f80-7d7de4494b00","Type":"ContainerStarted","Data":"c97541349507ffb3384640e891e191106c5387c58b0946edba6fc2bc290f3ec6"}
Oct 30 19:49:50.069096 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.069025    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7"
Oct 30 19:49:50.113400 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.113377    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"
Oct 30 19:49:50.118183 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.118145    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8" podStartSLOduration=1.657132732 podStartE2EDuration="2.118130084s" totalImagesPullingTime="460.997352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:50.092782084 +0000 UTC m=+1144.677051702" watchObservedRunningTime="2026-10-30 19:49:50.118130084 +0000 UTC m=+1144.702399725"
Oct 30 19:49:50.118784 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.118759    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7" podStartSLOduration=1.727994678 podStartE2EDuration="4.118750672s" totalImagesPullingTime="2.390755994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:50.117292536 +0000 UTC m=+1144.701562154" watchObservedRunningTime="2026-10-30 19:49:50.118750672 +0000 UTC m=+1144.703020290"
Oct 30 19:49:50.394216 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.394188    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"]
Oct 30 19:49:50.396816 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:49:50.396788    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded52fa8d_f9f6_4b79_bdf5_32af6f7b9b51.slice/crio-f6da300af388c88f79b2b7c507bc58261e6b50f5081759340d1fd397a63e0a25 WatchSource:0}: Error finding container f6da300af388c88f79b2b7c507bc58261e6b50f5081759340d1fd397a63e0a25: Status 404 returned error can't find the container with id f6da300af388c88f79b2b7c507bc58261e6b50f5081759340d1fd397a63e0a25
Oct 30 19:49:50.793553 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.793512    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"]
Oct 30 19:49:50.797957 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.797936    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"
Oct 30 19:49:50.810841 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.810786    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-a1d885-serving-cert\""
Oct 30 19:49:50.811405 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.811383    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"]
Oct 30 19:49:50.943758 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.943725    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c8ac1e-8cf9-4a86-affa-9a2745709749-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk\" (UID: \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"
Oct 30 19:49:50.943758 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.943757    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmq8d\" (UniqueName: \"kubernetes.io/projected/d8c8ac1e-8cf9-4a86-affa-9a2745709749-kube-api-access-bmq8d\") pod \"maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk\" (UID: \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"
Oct 30 19:49:50.943983 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.943806    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c8ac1e-8cf9-4a86-affa-9a2745709749-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk\" (UID: \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"
Oct 30 19:49:50.944078 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:50.944044    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c8ac1e-8cf9-4a86-affa-9a2745709749-metrics-certs\") pod \"maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk\" (UID: \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"
Oct 30 19:49:51.045361 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.045271    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c8ac1e-8cf9-4a86-affa-9a2745709749-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk\" (UID: \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"
Oct 30 19:49:51.045530 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.045359    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmq8d\" (UniqueName: \"kubernetes.io/projected/d8c8ac1e-8cf9-4a86-affa-9a2745709749-kube-api-access-bmq8d\") pod \"maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk\" (UID: \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"
Oct 30 19:49:51.045530 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.045403    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c8ac1e-8cf9-4a86-affa-9a2745709749-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk\" (UID: \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"
Oct 30 19:49:51.045530 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.045485    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c8ac1e-8cf9-4a86-affa-9a2745709749-metrics-certs\") pod \"maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk\" (UID: \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"
Oct 30 19:49:51.045970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.045940    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c8ac1e-8cf9-4a86-affa-9a2745709749-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk\" (UID: \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"
Oct 30 19:49:51.048006 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.047980    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c8ac1e-8cf9-4a86-affa-9a2745709749-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk\" (UID: \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"
Oct 30 19:49:51.048266 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.048248    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c8ac1e-8cf9-4a86-affa-9a2745709749-metrics-certs\") pod \"maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk\" (UID: \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"
Oct 30 19:49:51.059848 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.059823    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmq8d\" (UniqueName: \"kubernetes.io/projected/d8c8ac1e-8cf9-4a86-affa-9a2745709749-kube-api-access-bmq8d\") pod \"maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk\" (UID: \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"
Oct 30 19:49:51.075420 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.075394    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx" event={"ID":"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51","Type":"ContainerStarted","Data":"2e009b201eefc50928bf0987e06f76d98868f454d4a3748229a01c9cb9ae34f5"}
Oct 30 19:49:51.075525 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.075450    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx" event={"ID":"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51","Type":"ContainerStarted","Data":"f6da300af388c88f79b2b7c507bc58261e6b50f5081759340d1fd397a63e0a25"}
Oct 30 19:49:51.075525 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.075501    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"
Oct 30 19:49:51.111569 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.111549    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"
Oct 30 19:49:51.387590 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.387494    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx" podStartSLOduration=1.972141706 podStartE2EDuration="2.387479878s" totalImagesPullingTime="415.338172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:51.10147527 +0000 UTC m=+1145.685744900" watchObservedRunningTime="2026-10-30 19:49:51.387479878 +0000 UTC m=+1145.971749492"
Oct 30 19:49:51.390443 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.390406    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"]
Oct 30 19:49:51.398441 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:49:51.398415    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c8ac1e_8cf9_4a86_affa_9a2745709749.slice/crio-56d5399430f0894a9b47fd0699f3b498a18dedd372f50fa7f0967196cdc790da WatchSource:0}: Error finding container 56d5399430f0894a9b47fd0699f3b498a18dedd372f50fa7f0967196cdc790da: Status 404 returned error can't find the container with id 56d5399430f0894a9b47fd0699f3b498a18dedd372f50fa7f0967196cdc790da
Oct 30 19:49:51.837250 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.837207    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"]
Oct 30 19:49:51.841976 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.841954    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"
Oct 30 19:49:51.874178 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.874147    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"]
Oct 30 19:49:51.874349 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.874317    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-ced832-serving-cert\""
Oct 30 19:49:51.952754 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.952726    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66fb1cac-4e6d-4d5a-812d-5054e1676769-metrics-certs\") pod \"maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd\" (UID: \"66fb1cac-4e6d-4d5a-812d-5054e1676769\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"
Oct 30 19:49:51.952893 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.952760    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66fb1cac-4e6d-4d5a-812d-5054e1676769-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd\" (UID: \"66fb1cac-4e6d-4d5a-812d-5054e1676769\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"
Oct 30 19:49:51.952893 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.952793    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj8pb\" (UniqueName: \"kubernetes.io/projected/66fb1cac-4e6d-4d5a-812d-5054e1676769-kube-api-access-nj8pb\") pod \"maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd\" (UID: \"66fb1cac-4e6d-4d5a-812d-5054e1676769\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"
Oct 30 19:49:51.952981 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:51.952961    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66fb1cac-4e6d-4d5a-812d-5054e1676769-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd\" (UID: \"66fb1cac-4e6d-4d5a-812d-5054e1676769\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"
Oct 30 19:49:52.053440 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:52.053413    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"
Oct 30 19:49:52.053576 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:52.053553    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66fb1cac-4e6d-4d5a-812d-5054e1676769-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd\" (UID: \"66fb1cac-4e6d-4d5a-812d-5054e1676769\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"
Oct 30 19:49:52.053626 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:52.053606    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj8pb\" (UniqueName: \"kubernetes.io/projected/66fb1cac-4e6d-4d5a-812d-5054e1676769-kube-api-access-nj8pb\") pod \"maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd\" (UID: \"66fb1cac-4e6d-4d5a-812d-5054e1676769\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"
Oct 30 19:49:52.053697 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:52.053684    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66fb1cac-4e6d-4d5a-812d-5054e1676769-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd\" (UID: \"66fb1cac-4e6d-4d5a-812d-5054e1676769\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"
Oct 30 19:49:52.053769 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:52.053758    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66fb1cac-4e6d-4d5a-812d-5054e1676769-metrics-certs\") pod \"maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd\" (UID: \"66fb1cac-4e6d-4d5a-812d-5054e1676769\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"
Oct 30 19:49:52.054367 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:52.054345    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66fb1cac-4e6d-4d5a-812d-5054e1676769-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd\" (UID: \"66fb1cac-4e6d-4d5a-812d-5054e1676769\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"
Oct 30 19:49:52.055881 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:52.055861    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66fb1cac-4e6d-4d5a-812d-5054e1676769-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd\" (UID: \"66fb1cac-4e6d-4d5a-812d-5054e1676769\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"
Oct 30 19:49:52.055952 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:52.055903    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66fb1cac-4e6d-4d5a-812d-5054e1676769-metrics-certs\") pod \"maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd\" (UID: \"66fb1cac-4e6d-4d5a-812d-5054e1676769\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"
Oct 30 19:49:52.070920 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:52.070901    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj8pb\" (UniqueName: \"kubernetes.io/projected/66fb1cac-4e6d-4d5a-812d-5054e1676769-kube-api-access-nj8pb\") pod \"maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd\" (UID: \"66fb1cac-4e6d-4d5a-812d-5054e1676769\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"
Oct 30 19:49:52.080197 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:52.080174    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk" event={"ID":"d8c8ac1e-8cf9-4a86-affa-9a2745709749","Type":"ContainerStarted","Data":"56d5399430f0894a9b47fd0699f3b498a18dedd372f50fa7f0967196cdc790da"}
Oct 30 19:49:52.154522 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:52.154495    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"
Oct 30 19:49:52.433520 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:52.433444    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"]
Oct 30 19:49:52.436847 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:49:52.436821    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66fb1cac_4e6d_4d5a_812d_5054e1676769.slice/crio-639d2e26d7c2a2f8e08ef6393c5e1db11efd0e892de54f53afa801a14389e1c9 WatchSource:0}: Error finding container 639d2e26d7c2a2f8e08ef6393c5e1db11efd0e892de54f53afa801a14389e1c9: Status 404 returned error can't find the container with id 639d2e26d7c2a2f8e08ef6393c5e1db11efd0e892de54f53afa801a14389e1c9
Oct 30 19:49:53.086342 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:53.086280    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd" event={"ID":"66fb1cac-4e6d-4d5a-812d-5054e1676769","Type":"ContainerStarted","Data":"b185cd94638e9903bebc2f25f6c7f8bc5d92c3a163b1f399bbef3844a9aa88c9"}
Oct 30 19:49:53.087085 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:53.087067    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"
Oct 30 19:49:53.087192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:53.087104    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd" event={"ID":"66fb1cac-4e6d-4d5a-812d-5054e1676769","Type":"ContainerStarted","Data":"639d2e26d7c2a2f8e08ef6393c5e1db11efd0e892de54f53afa801a14389e1c9"}
Oct 30 19:49:53.088018 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:53.087997    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk" event={"ID":"d8c8ac1e-8cf9-4a86-affa-9a2745709749","Type":"ContainerStarted","Data":"1ad6f6cb2c0f9f5b2833fb6daccc0266b357a446399418b0600eb760b01167ca"}
Oct 30 19:49:53.088167 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:53.088155    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"
Oct 30 19:49:53.135647 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:53.135582    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd" podStartSLOduration=1.587992093 podStartE2EDuration="2.135569855s" totalImagesPullingTime="547.577762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:53.13541822 +0000 UTC m=+1147.719687842" watchObservedRunningTime="2026-10-30 19:49:53.135569855 +0000 UTC m=+1147.719839472"
Oct 30 19:49:53.164018 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:53.163964    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk" podStartSLOduration=2.5350561369999998 podStartE2EDuration="3.163948277s" totalImagesPullingTime="628.89214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:53.162979796 +0000 UTC m=+1147.747249412" watchObservedRunningTime="2026-10-30 19:49:53.163948277 +0000 UTC m=+1147.748217895"
Oct 30 19:49:56.081293 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.081265    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"
Oct 30 19:49:56.323410 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.323377    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"]
Oct 30 19:49:56.327344 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.327322    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"
Oct 30 19:49:56.336321 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.336301    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-5f933e-serving-cert\""
Oct 30 19:49:56.351272 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.351241    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"]
Oct 30 19:49:56.503234 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.503206    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b518877-72bf-4743-8f2e-daa5f9800392-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj\" (UID: \"5b518877-72bf-4743-8f2e-daa5f9800392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"
Oct 30 19:49:56.503420 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.503243    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b518877-72bf-4743-8f2e-daa5f9800392-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj\" (UID: \"5b518877-72bf-4743-8f2e-daa5f9800392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"
Oct 30 19:49:56.503420 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.503394    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx4sb\" (UniqueName: \"kubernetes.io/projected/5b518877-72bf-4743-8f2e-daa5f9800392-kube-api-access-vx4sb\") pod \"maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj\" (UID: \"5b518877-72bf-4743-8f2e-daa5f9800392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"
Oct 30 19:49:56.503544 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.503444    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b518877-72bf-4743-8f2e-daa5f9800392-metrics-certs\") pod \"maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj\" (UID: \"5b518877-72bf-4743-8f2e-daa5f9800392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"
Oct 30 19:49:56.604845 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.604758    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b518877-72bf-4743-8f2e-daa5f9800392-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj\" (UID: \"5b518877-72bf-4743-8f2e-daa5f9800392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"
Oct 30 19:49:56.604845 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.604806    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b518877-72bf-4743-8f2e-daa5f9800392-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj\" (UID: \"5b518877-72bf-4743-8f2e-daa5f9800392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"
Oct 30 19:49:56.605115 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.604886    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx4sb\" (UniqueName: \"kubernetes.io/projected/5b518877-72bf-4743-8f2e-daa5f9800392-kube-api-access-vx4sb\") pod \"maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj\" (UID: \"5b518877-72bf-4743-8f2e-daa5f9800392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"
Oct 30 19:49:56.605115 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.604924    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b518877-72bf-4743-8f2e-daa5f9800392-metrics-certs\") pod \"maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj\" (UID: \"5b518877-72bf-4743-8f2e-daa5f9800392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"
Oct 30 19:49:56.605452 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.605430    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b518877-72bf-4743-8f2e-daa5f9800392-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj\" (UID: \"5b518877-72bf-4743-8f2e-daa5f9800392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"
Oct 30 19:49:56.607368 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.607344    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b518877-72bf-4743-8f2e-daa5f9800392-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj\" (UID: \"5b518877-72bf-4743-8f2e-daa5f9800392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"
Oct 30 19:49:56.607535 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.607516    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b518877-72bf-4743-8f2e-daa5f9800392-metrics-certs\") pod \"maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj\" (UID: \"5b518877-72bf-4743-8f2e-daa5f9800392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"
Oct 30 19:49:56.618090 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.618069    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx4sb\" (UniqueName: \"kubernetes.io/projected/5b518877-72bf-4743-8f2e-daa5f9800392-kube-api-access-vx4sb\") pod \"maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj\" (UID: \"5b518877-72bf-4743-8f2e-daa5f9800392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"
Oct 30 19:49:56.640911 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.640887    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"
Oct 30 19:49:56.927586 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:56.927276    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"]
Oct 30 19:49:56.932984 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:49:56.932948    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b518877_72bf_4743_8f2e_daa5f9800392.slice/crio-d94a8e99192ca159b14aec6eebe481fc1c4181c097c89806613cbce13b3b0932 WatchSource:0}: Error finding container d94a8e99192ca159b14aec6eebe481fc1c4181c097c89806613cbce13b3b0932: Status 404 returned error can't find the container with id d94a8e99192ca159b14aec6eebe481fc1c4181c097c89806613cbce13b3b0932
Oct 30 19:49:57.086135 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:57.086099    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"
Oct 30 19:49:57.107198 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:57.107172    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj" event={"ID":"5b518877-72bf-4743-8f2e-daa5f9800392","Type":"ContainerStarted","Data":"d94a8e99192ca159b14aec6eebe481fc1c4181c097c89806613cbce13b3b0932"}
Oct 30 19:49:58.116779 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:58.116740    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj" event={"ID":"5b518877-72bf-4743-8f2e-daa5f9800392","Type":"ContainerStarted","Data":"7f950bd8970eac766cc35fdcfc09df95aea64da2e84931504bca9f8ff0861ccc"}
Oct 30 19:49:58.117431 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:58.117411    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"
Oct 30 19:49:58.154528 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:58.154464    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj" podStartSLOduration=1.7423035009999999 podStartE2EDuration="2.154445131s" totalImagesPullingTime="412.14163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:58.151466164 +0000 UTC m=+1152.735735781" watchObservedRunningTime="2026-10-30 19:49:58.154445131 +0000 UTC m=+1152.738714751"
Oct 30 19:49:59.098181 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:59.098151    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"
Oct 30 19:49:59.098620 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:49:59.098598    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"
Oct 30 19:50:01.078694 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:01.078669    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7"
Oct 30 19:50:04.127403 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:04.127372    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"
Oct 30 19:50:09.659215 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:50:09.659104    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:50:10.959167 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:10.959126    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"]
Oct 30 19:50:10.964719 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:10.964701    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:10.984280 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:10.984250    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"]
Oct 30 19:50:10.988366 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:10.988333    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ced832\"/\"kube-root-ca.crt\""
Oct 30 19:50:10.988485 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:10.988440    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ced832\"/\"openshift-service-ca.crt\""
Oct 30 19:50:10.988581 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:10.988558    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ced832\"/\"simulator-w0-ced832-kserve-self-signed-certs\""
Oct 30 19:50:10.988681 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:10.988637    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ced832\"/\"default-dockercfg-vjsbn\""
Oct 30 19:50:11.062389 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.062343    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-dshm\") pod \"simulator-w0-ced832-kserve-86c4d4968d-9tfqh\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") " pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.062577 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.062411    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9c5h\" (UniqueName: \"kubernetes.io/projected/f475ddeb-6235-4af9-a57c-7c4f5160fda9-kube-api-access-h9c5h\") pod \"simulator-w0-ced832-kserve-86c4d4968d-9tfqh\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") " pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.062577 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.062469    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-tmp-dir\") pod \"simulator-w0-ced832-kserve-86c4d4968d-9tfqh\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") " pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.062577 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.062548    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f475ddeb-6235-4af9-a57c-7c4f5160fda9-tls-certs\") pod \"simulator-w0-ced832-kserve-86c4d4968d-9tfqh\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") " pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.062717 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.062692    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-model-cache\") pod \"simulator-w0-ced832-kserve-86c4d4968d-9tfqh\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") " pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.062763 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.062747    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-home\") pod \"simulator-w0-ced832-kserve-86c4d4968d-9tfqh\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") " pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.164303 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.164262    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f475ddeb-6235-4af9-a57c-7c4f5160fda9-tls-certs\") pod \"simulator-w0-ced832-kserve-86c4d4968d-9tfqh\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") " pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.164499 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.164328    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-model-cache\") pod \"simulator-w0-ced832-kserve-86c4d4968d-9tfqh\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") " pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.164499 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.164362    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-home\") pod \"simulator-w0-ced832-kserve-86c4d4968d-9tfqh\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") " pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.164499 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.164420    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-dshm\") pod \"simulator-w0-ced832-kserve-86c4d4968d-9tfqh\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") " pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.164499 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.164458    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9c5h\" (UniqueName: \"kubernetes.io/projected/f475ddeb-6235-4af9-a57c-7c4f5160fda9-kube-api-access-h9c5h\") pod \"simulator-w0-ced832-kserve-86c4d4968d-9tfqh\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") " pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.164499 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.164485    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-tmp-dir\") pod \"simulator-w0-ced832-kserve-86c4d4968d-9tfqh\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") " pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.164800 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.164773    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-model-cache\") pod \"simulator-w0-ced832-kserve-86c4d4968d-9tfqh\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") " pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.164924 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.164829    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-home\") pod \"simulator-w0-ced832-kserve-86c4d4968d-9tfqh\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") " pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.164924 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.164895    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-tmp-dir\") pod \"simulator-w0-ced832-kserve-86c4d4968d-9tfqh\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") " pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.167067 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.167025    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-dshm\") pod \"simulator-w0-ced832-kserve-86c4d4968d-9tfqh\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") " pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.167627 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.167547    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f475ddeb-6235-4af9-a57c-7c4f5160fda9-tls-certs\") pod \"simulator-w0-ced832-kserve-86c4d4968d-9tfqh\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") " pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.188844 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.188810    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9c5h\" (UniqueName: \"kubernetes.io/projected/f475ddeb-6235-4af9-a57c-7c4f5160fda9-kube-api-access-h9c5h\") pod \"simulator-w0-ced832-kserve-86c4d4968d-9tfqh\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") " pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.276615 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.276579    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:11.567931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:11.567850    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"]
Oct 30 19:50:11.571468 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:50:11.571445    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf475ddeb_6235_4af9_a57c_7c4f5160fda9.slice/crio-1bf2a83d4778861559b9207c9c151c4c035322f46db58a44410139300e15f0c2 WatchSource:0}: Error finding container 1bf2a83d4778861559b9207c9c151c4c035322f46db58a44410139300e15f0c2: Status 404 returned error can't find the container with id 1bf2a83d4778861559b9207c9c151c4c035322f46db58a44410139300e15f0c2
Oct 30 19:50:12.201168 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:12.201128    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" event={"ID":"f475ddeb-6235-4af9-a57c-7c4f5160fda9","Type":"ContainerStarted","Data":"1bf2a83d4778861559b9207c9c151c4c035322f46db58a44410139300e15f0c2"}
Oct 30 19:50:14.213898 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:14.213856    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" event={"ID":"f475ddeb-6235-4af9-a57c-7c4f5160fda9","Type":"ContainerStarted","Data":"bb45b7251f3e1c7868218bd20f939a72f3b4ca5c7dfa48ef21ca065c71857ba3"}
Oct 30 19:50:14.248506 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:14.248457    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podStartSLOduration=2.549504999 podStartE2EDuration="4.2484429s" totalImagesPullingTime="1.698937901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:14.246153414 +0000 UTC m=+1168.830423032" watchObservedRunningTime="2026-10-30 19:50:14.2484429 +0000 UTC m=+1168.832712520"
Oct 30 19:50:14.276867 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:14.276835    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:21.277221 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:21.277182    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:21.289451 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:21.289429    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:22.178290 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.178258    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"]
Oct 30 19:50:22.182893 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.182873    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.188343 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.188318    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-278ffa\"/\"openshift-service-ca.crt\""
Oct 30 19:50:22.189724 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.189700    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-278ffa\"/\"kube-root-ca.crt\""
Oct 30 19:50:22.189855 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.189762    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-278ffa\"/\"simulator-w5-278ffa-kserve-self-signed-certs\""
Oct 30 19:50:22.189855 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.189820    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-278ffa\"/\"default-dockercfg-2t28t\""
Oct 30 19:50:22.204547 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.204520    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"]
Oct 30 19:50:22.262108 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.262079    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:50:22.275056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.275011    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-model-cache\") pod \"simulator-w5-278ffa-kserve-66c45dc6d-59l8q\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") " pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.275197 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.275090    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-dshm\") pod \"simulator-w5-278ffa-kserve-66c45dc6d-59l8q\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") " pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.275258 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.275190    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttfkc\" (UniqueName: \"kubernetes.io/projected/910fde4a-f4d1-4547-a060-2cf6888e5942-kube-api-access-ttfkc\") pod \"simulator-w5-278ffa-kserve-66c45dc6d-59l8q\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") " pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.275258 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.275236    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-home\") pod \"simulator-w5-278ffa-kserve-66c45dc6d-59l8q\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") " pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.275377 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.275359    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-tmp-dir\") pod \"simulator-w5-278ffa-kserve-66c45dc6d-59l8q\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") " pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.275448 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.275433    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910fde4a-f4d1-4547-a060-2cf6888e5942-tls-certs\") pod \"simulator-w5-278ffa-kserve-66c45dc6d-59l8q\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") " pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.376088 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.376059    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-tmp-dir\") pod \"simulator-w5-278ffa-kserve-66c45dc6d-59l8q\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") " pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.376538 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.376116    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910fde4a-f4d1-4547-a060-2cf6888e5942-tls-certs\") pod \"simulator-w5-278ffa-kserve-66c45dc6d-59l8q\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") " pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.376538 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.376171    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-model-cache\") pod \"simulator-w5-278ffa-kserve-66c45dc6d-59l8q\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") " pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.376538 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.376204    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-dshm\") pod \"simulator-w5-278ffa-kserve-66c45dc6d-59l8q\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") " pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.376538 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.376252    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttfkc\" (UniqueName: \"kubernetes.io/projected/910fde4a-f4d1-4547-a060-2cf6888e5942-kube-api-access-ttfkc\") pod \"simulator-w5-278ffa-kserve-66c45dc6d-59l8q\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") " pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.376538 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.376287    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-home\") pod \"simulator-w5-278ffa-kserve-66c45dc6d-59l8q\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") " pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.376818 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.376743    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-model-cache\") pod \"simulator-w5-278ffa-kserve-66c45dc6d-59l8q\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") " pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.376818 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.376793    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-home\") pod \"simulator-w5-278ffa-kserve-66c45dc6d-59l8q\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") " pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.376908 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.376855    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-tmp-dir\") pod \"simulator-w5-278ffa-kserve-66c45dc6d-59l8q\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") " pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.378812 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.378789    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910fde4a-f4d1-4547-a060-2cf6888e5942-tls-certs\") pod \"simulator-w5-278ffa-kserve-66c45dc6d-59l8q\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") " pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.378935 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.378800    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-dshm\") pod \"simulator-w5-278ffa-kserve-66c45dc6d-59l8q\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") " pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.388988 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.388964    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttfkc\" (UniqueName: \"kubernetes.io/projected/910fde4a-f4d1-4547-a060-2cf6888e5942-kube-api-access-ttfkc\") pod \"simulator-w5-278ffa-kserve-66c45dc6d-59l8q\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") " pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.494833 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.494792    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:22.776053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:22.776010    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"]
Oct 30 19:50:22.778584 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:50:22.778547    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod910fde4a_f4d1_4547_a060_2cf6888e5942.slice/crio-a55083a43be6dee80e680b30512e1cd4cece151db8f99ad87bcf3d95daa081de WatchSource:0}: Error finding container a55083a43be6dee80e680b30512e1cd4cece151db8f99ad87bcf3d95daa081de: Status 404 returned error can't find the container with id a55083a43be6dee80e680b30512e1cd4cece151db8f99ad87bcf3d95daa081de
Oct 30 19:50:23.257705 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:23.257667    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" event={"ID":"910fde4a-f4d1-4547-a060-2cf6888e5942","Type":"ContainerStarted","Data":"736ccf33812f9fa040dfe650c49a85a2e917017a25af0537e65da7e1547bd8aa"}
Oct 30 19:50:23.257943 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:23.257925    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" event={"ID":"910fde4a-f4d1-4547-a060-2cf6888e5942","Type":"ContainerStarted","Data":"a55083a43be6dee80e680b30512e1cd4cece151db8f99ad87bcf3d95daa081de"}
Oct 30 19:50:23.284809 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:23.284753    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podStartSLOduration=1.284736356 podStartE2EDuration="1.284736356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:23.282532389 +0000 UTC m=+1177.866802007" watchObservedRunningTime="2026-10-30 19:50:23.284736356 +0000 UTC m=+1177.869005973"
Oct 30 19:50:23.495887 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:23.495858    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:32.495695 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:32.495613    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:32.508695 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:32.508671    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:33.318004 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:33.317975    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:50:39.056544 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.056512    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"]
Oct 30 19:50:39.060113 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.060092    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.068515 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.068486    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"openshift-service-ca.crt\""
Oct 30 19:50:39.068677 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.068659    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"kube-root-ca.crt\""
Oct 30 19:50:39.070022 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.070002    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"premium-w1-a1d885-kserve-self-signed-certs\""
Oct 30 19:50:39.070022 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.070012    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"default-dockercfg-fzr2x\""
Oct 30 19:50:39.073580 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.073559    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"]
Oct 30 19:50:39.251819 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.250541    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-tmp-dir\") pod \"premium-w1-a1d885-kserve-5b494dbc9d-bjj9t\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") " pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.251819 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.250599    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvmrz\" (UniqueName: \"kubernetes.io/projected/f141028a-4c85-40ca-b0aa-cdc0838170d1-kube-api-access-qvmrz\") pod \"premium-w1-a1d885-kserve-5b494dbc9d-bjj9t\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") " pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.251819 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.250641    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-dshm\") pod \"premium-w1-a1d885-kserve-5b494dbc9d-bjj9t\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") " pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.251819 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.250674    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-home\") pod \"premium-w1-a1d885-kserve-5b494dbc9d-bjj9t\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") " pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.251819 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.250777    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-model-cache\") pod \"premium-w1-a1d885-kserve-5b494dbc9d-bjj9t\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") " pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.251819 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.250842    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f141028a-4c85-40ca-b0aa-cdc0838170d1-tls-certs\") pod \"premium-w1-a1d885-kserve-5b494dbc9d-bjj9t\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") " pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.352078 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.351959    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-dshm\") pod \"premium-w1-a1d885-kserve-5b494dbc9d-bjj9t\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") " pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.352078 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.352014    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-home\") pod \"premium-w1-a1d885-kserve-5b494dbc9d-bjj9t\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") " pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.352316 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.352133    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-model-cache\") pod \"premium-w1-a1d885-kserve-5b494dbc9d-bjj9t\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") " pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.352316 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.352169    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f141028a-4c85-40ca-b0aa-cdc0838170d1-tls-certs\") pod \"premium-w1-a1d885-kserve-5b494dbc9d-bjj9t\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") " pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.352316 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.352232    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-tmp-dir\") pod \"premium-w1-a1d885-kserve-5b494dbc9d-bjj9t\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") " pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.352316 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.352259    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmrz\" (UniqueName: \"kubernetes.io/projected/f141028a-4c85-40ca-b0aa-cdc0838170d1-kube-api-access-qvmrz\") pod \"premium-w1-a1d885-kserve-5b494dbc9d-bjj9t\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") " pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.352596 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.352568    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-model-cache\") pod \"premium-w1-a1d885-kserve-5b494dbc9d-bjj9t\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") " pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.352596 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.352581    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-tmp-dir\") pod \"premium-w1-a1d885-kserve-5b494dbc9d-bjj9t\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") " pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.352724 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.352664    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-home\") pod \"premium-w1-a1d885-kserve-5b494dbc9d-bjj9t\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") " pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.355149 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.355125    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-dshm\") pod \"premium-w1-a1d885-kserve-5b494dbc9d-bjj9t\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") " pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.355650 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.355630    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f141028a-4c85-40ca-b0aa-cdc0838170d1-tls-certs\") pod \"premium-w1-a1d885-kserve-5b494dbc9d-bjj9t\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") " pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.365721 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.365691    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvmrz\" (UniqueName: \"kubernetes.io/projected/f141028a-4c85-40ca-b0aa-cdc0838170d1-kube-api-access-qvmrz\") pod \"premium-w1-a1d885-kserve-5b494dbc9d-bjj9t\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") " pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.374469 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.374446    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:39.660397 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:39.660319    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"]
Oct 30 19:50:39.663657 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:50:39.663619    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf141028a_4c85_40ca_b0aa_cdc0838170d1.slice/crio-0837f479a2f75570d54ffa23901feebedc0060a046008a04c0932b53eea102f9 WatchSource:0}: Error finding container 0837f479a2f75570d54ffa23901feebedc0060a046008a04c0932b53eea102f9: Status 404 returned error can't find the container with id 0837f479a2f75570d54ffa23901feebedc0060a046008a04c0932b53eea102f9
Oct 30 19:50:39.663784 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:50:39.663684    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:50:40.341481 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:40.341448    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" event={"ID":"f141028a-4c85-40ca-b0aa-cdc0838170d1","Type":"ContainerStarted","Data":"84846debeba4483c62fe23687f46f9462f043eabeddb23f07df2c7fe6cabc40e"}
Oct 30 19:50:40.341945 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:40.341489    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" event={"ID":"f141028a-4c85-40ca-b0aa-cdc0838170d1","Type":"ContainerStarted","Data":"0837f479a2f75570d54ffa23901feebedc0060a046008a04c0932b53eea102f9"}
Oct 30 19:50:40.367507 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:40.367463    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podStartSLOduration=1.367450398 podStartE2EDuration="1.367450398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:40.365618635 +0000 UTC m=+1194.949888254" watchObservedRunningTime="2026-10-30 19:50:40.367450398 +0000 UTC m=+1194.951720060"
Oct 30 19:50:40.375144 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:40.375123    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:49.374939 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:49.374900    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:49.388337 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:49.388311    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:49.399833 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:49.399811    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:50:55.730455 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.730420    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"]
Oct 30 19:50:55.735744 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.735717    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:55.757173 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.757150    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-278ffa\"/\"premium-w5-278ffa-kserve-self-signed-certs\""
Oct 30 19:50:55.758433 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.758413    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"]
Oct 30 19:50:55.863713 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.863666    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-tmp-dir\") pod \"premium-w5-278ffa-kserve-f4f68c54f-89s58\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") " pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:55.864015 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.863993    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c65275-2119-45ac-8393-c29bce507f33-tls-certs\") pod \"premium-w5-278ffa-kserve-f4f68c54f-89s58\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") " pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:55.864264 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.864243    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqf9g\" (UniqueName: \"kubernetes.io/projected/99c65275-2119-45ac-8393-c29bce507f33-kube-api-access-sqf9g\") pod \"premium-w5-278ffa-kserve-f4f68c54f-89s58\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") " pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:55.864403 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.864389    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-model-cache\") pod \"premium-w5-278ffa-kserve-f4f68c54f-89s58\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") " pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:55.864527 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.864513    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-home\") pod \"premium-w5-278ffa-kserve-f4f68c54f-89s58\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") " pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:55.864625 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.864613    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-dshm\") pod \"premium-w5-278ffa-kserve-f4f68c54f-89s58\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") " pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:55.966157 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.966118    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-home\") pod \"premium-w5-278ffa-kserve-f4f68c54f-89s58\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") " pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:55.966157 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.966159    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-dshm\") pod \"premium-w5-278ffa-kserve-f4f68c54f-89s58\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") " pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:55.966406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.966259    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-tmp-dir\") pod \"premium-w5-278ffa-kserve-f4f68c54f-89s58\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") " pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:55.966406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.966292    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c65275-2119-45ac-8393-c29bce507f33-tls-certs\") pod \"premium-w5-278ffa-kserve-f4f68c54f-89s58\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") " pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:55.966406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.966332    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqf9g\" (UniqueName: \"kubernetes.io/projected/99c65275-2119-45ac-8393-c29bce507f33-kube-api-access-sqf9g\") pod \"premium-w5-278ffa-kserve-f4f68c54f-89s58\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") " pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:55.966406 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.966386    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-model-cache\") pod \"premium-w5-278ffa-kserve-f4f68c54f-89s58\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") " pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:55.966654 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.966628    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-home\") pod \"premium-w5-278ffa-kserve-f4f68c54f-89s58\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") " pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:55.966725 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.966702    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-tmp-dir\") pod \"premium-w5-278ffa-kserve-f4f68c54f-89s58\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") " pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:55.966785 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.966750    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-model-cache\") pod \"premium-w5-278ffa-kserve-f4f68c54f-89s58\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") " pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:55.968557 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.968531    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-dshm\") pod \"premium-w5-278ffa-kserve-f4f68c54f-89s58\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") " pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:55.968830 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.968810    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c65275-2119-45ac-8393-c29bce507f33-tls-certs\") pod \"premium-w5-278ffa-kserve-f4f68c54f-89s58\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") " pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:55.979540 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:55.979518    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqf9g\" (UniqueName: \"kubernetes.io/projected/99c65275-2119-45ac-8393-c29bce507f33-kube-api-access-sqf9g\") pod \"premium-w5-278ffa-kserve-f4f68c54f-89s58\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") " pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:56.050475 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:56.050440    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:50:56.331758 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:56.331679    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"]
Oct 30 19:50:56.335122 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:50:56.335097    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99c65275_2119_45ac_8393_c29bce507f33.slice/crio-2c7eb4b28af1c173410ab0a49563e48ff17f66b665f42b73944c20a84f8a63fc WatchSource:0}: Error finding container 2c7eb4b28af1c173410ab0a49563e48ff17f66b665f42b73944c20a84f8a63fc: Status 404 returned error can't find the container with id 2c7eb4b28af1c173410ab0a49563e48ff17f66b665f42b73944c20a84f8a63fc
Oct 30 19:50:56.431706 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:56.431668    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" event={"ID":"99c65275-2119-45ac-8393-c29bce507f33","Type":"ContainerStarted","Data":"5296966643fd29d112dcc525a53ef5a52c7e84bb6ed167fd2cad3d72808230c8"}
Oct 30 19:50:56.431904 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:56.431721    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" event={"ID":"99c65275-2119-45ac-8393-c29bce507f33","Type":"ContainerStarted","Data":"2c7eb4b28af1c173410ab0a49563e48ff17f66b665f42b73944c20a84f8a63fc"}
Oct 30 19:50:56.482995 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:56.482943    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podStartSLOduration=1.482927802 podStartE2EDuration="1.482927802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:56.482554575 +0000 UTC m=+1211.066824192" watchObservedRunningTime="2026-10-30 19:50:56.482927802 +0000 UTC m=+1211.067197469"
Oct 30 19:50:57.051297 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:50:57.051250    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:51:06.051411 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:06.051374    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:51:06.063897 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:06.063872    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:51:06.491063 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:06.491018    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:51:09.659747 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:51:09.659703    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:51:39.659380 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:51:39.659341    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:51:56.361654 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.361616    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"]
Oct 30 19:51:56.366077 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.366058    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.371656 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.371632    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ced832\"/\"distinct-w0-ced832-kserve-self-signed-certs\""
Oct 30 19:51:56.377106 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.377082    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"]
Oct 30 19:51:56.402001 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.401965    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-dshm\") pod \"distinct-w0-ced832-kserve-7f9c4956d-25ncs\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.402164 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.402092    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca13cd9-66b8-456e-8147-5a951cc47d88-tls-certs\") pod \"distinct-w0-ced832-kserve-7f9c4956d-25ncs\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.402164 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.402122    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-model-cache\") pod \"distinct-w0-ced832-kserve-7f9c4956d-25ncs\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.402281 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.402175    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-tmp-dir\") pod \"distinct-w0-ced832-kserve-7f9c4956d-25ncs\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.402281 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.402225    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frqn8\" (UniqueName: \"kubernetes.io/projected/eca13cd9-66b8-456e-8147-5a951cc47d88-kube-api-access-frqn8\") pod \"distinct-w0-ced832-kserve-7f9c4956d-25ncs\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.402281 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.402263    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-home\") pod \"distinct-w0-ced832-kserve-7f9c4956d-25ncs\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.507786 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.507745    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-home\") pod \"distinct-w0-ced832-kserve-7f9c4956d-25ncs\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.508251 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.508225    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-dshm\") pod \"distinct-w0-ced832-kserve-7f9c4956d-25ncs\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.508490 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.508469    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca13cd9-66b8-456e-8147-5a951cc47d88-tls-certs\") pod \"distinct-w0-ced832-kserve-7f9c4956d-25ncs\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.508620 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.508605    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-model-cache\") pod \"distinct-w0-ced832-kserve-7f9c4956d-25ncs\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.508754 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.508740    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-tmp-dir\") pod \"distinct-w0-ced832-kserve-7f9c4956d-25ncs\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.508876 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.508862    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frqn8\" (UniqueName: \"kubernetes.io/projected/eca13cd9-66b8-456e-8147-5a951cc47d88-kube-api-access-frqn8\") pod \"distinct-w0-ced832-kserve-7f9c4956d-25ncs\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.509751 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.509728    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-home\") pod \"distinct-w0-ced832-kserve-7f9c4956d-25ncs\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.510378 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.510355    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-model-cache\") pod \"distinct-w0-ced832-kserve-7f9c4956d-25ncs\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.510817 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.510758    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-tmp-dir\") pod \"distinct-w0-ced832-kserve-7f9c4956d-25ncs\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.538050 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.527312    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca13cd9-66b8-456e-8147-5a951cc47d88-tls-certs\") pod \"distinct-w0-ced832-kserve-7f9c4956d-25ncs\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.538588 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.538440    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-dshm\") pod \"distinct-w0-ced832-kserve-7f9c4956d-25ncs\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.557986 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.557957    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frqn8\" (UniqueName: \"kubernetes.io/projected/eca13cd9-66b8-456e-8147-5a951cc47d88-kube-api-access-frqn8\") pod \"distinct-w0-ced832-kserve-7f9c4956d-25ncs\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:56.680093 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:56.680010    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:51:57.002469 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:57.002439    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"]
Oct 30 19:51:57.003652 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:51:57.003627    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeca13cd9_66b8_456e_8147_5a951cc47d88.slice/crio-309972040c11950ad992ba2f93216b082052ec853a32e37c69ae66f01a7c7c76 WatchSource:0}: Error finding container 309972040c11950ad992ba2f93216b082052ec853a32e37c69ae66f01a7c7c76: Status 404 returned error can't find the container with id 309972040c11950ad992ba2f93216b082052ec853a32e37c69ae66f01a7c7c76
Oct 30 19:51:57.757714 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:57.757685    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" event={"ID":"eca13cd9-66b8-456e-8147-5a951cc47d88","Type":"ContainerStarted","Data":"f028456ba5be02147eb8f1e3767153b3bd5bfb96d053c0cb7b43f4196280448d"}
Oct 30 19:51:57.758138 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:57.757722    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" event={"ID":"eca13cd9-66b8-456e-8147-5a951cc47d88","Type":"ContainerStarted","Data":"309972040c11950ad992ba2f93216b082052ec853a32e37c69ae66f01a7c7c76"}
Oct 30 19:51:57.798359 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:57.798311    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podStartSLOduration=1.798297803 podStartE2EDuration="1.798297803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:57.797938128 +0000 UTC m=+1272.382207744" watchObservedRunningTime="2026-10-30 19:51:57.798297803 +0000 UTC m=+1272.382567419"
Oct 30 19:51:58.680529 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:51:58.680497    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:52:06.681168 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:06.681132    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:52:06.693624 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:06.693600    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:52:06.810005 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:06.809974    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:52:09.659296 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:52:09.659261    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:52:10.061348 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.061292    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"]
Oct 30 19:52:10.065073 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.065045    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.078015 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.077994    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3f57c5-openshift-default-dockercfg-rshxj\""
Oct 30 19:52:10.089312 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.089274    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"]
Oct 30 19:52:10.181892 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.181854    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.182059 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.182041    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.182173 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.182147    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.182226 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.182203    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.182310 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.182290    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.182380 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.182326    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.182380 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.182353    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dwh4\" (UniqueName: \"kubernetes.io/projected/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-kube-api-access-2dwh4\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.182498 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.182423    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.182498 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.182465    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.239823 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.239792    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"]
Oct 30 19:52:10.243975 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.243957    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.253831 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.253804    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-278ffa\"/\"distinct-w5-278ffa-kserve-self-signed-certs\""
Oct 30 19:52:10.257991 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.257965    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"]
Oct 30 19:52:10.283786 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.283755    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.283913 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.283810    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.283913 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.283842    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.283913 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.283870    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.284105 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.283919    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.284105 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.283944    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dwh4\" (UniqueName: \"kubernetes.io/projected/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-kube-api-access-2dwh4\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.284105 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.283977    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.284105 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.284010    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.284105 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.284085    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.284345 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.284260    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.284420 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.284400    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.284590 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.284573    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.284686 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.284663    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.284896 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.284875    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.286991 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.286970    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.287433 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.287413    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.305681 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.305654    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dwh4\" (UniqueName: \"kubernetes.io/projected/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-kube-api-access-2dwh4\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.308146 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.308119    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.387320 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.387232    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:10.388808 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.388729    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbhd8\" (UniqueName: \"kubernetes.io/projected/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-kube-api-access-wbhd8\") pod \"distinct-w5-278ffa-kserve-5995c5f648-hfjm9\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.388957 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.388834    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-tmp-dir\") pod \"distinct-w5-278ffa-kserve-5995c5f648-hfjm9\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.388957 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.388886    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-dshm\") pod \"distinct-w5-278ffa-kserve-5995c5f648-hfjm9\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.388957 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.388929    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-tls-certs\") pod \"distinct-w5-278ffa-kserve-5995c5f648-hfjm9\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.389155 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.389009    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-model-cache\") pod \"distinct-w5-278ffa-kserve-5995c5f648-hfjm9\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.389155 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.389100    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-home\") pod \"distinct-w5-278ffa-kserve-5995c5f648-hfjm9\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.494804 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.490598    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-home\") pod \"distinct-w5-278ffa-kserve-5995c5f648-hfjm9\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.494804 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.490673    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbhd8\" (UniqueName: \"kubernetes.io/projected/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-kube-api-access-wbhd8\") pod \"distinct-w5-278ffa-kserve-5995c5f648-hfjm9\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.494804 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.490710    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-tmp-dir\") pod \"distinct-w5-278ffa-kserve-5995c5f648-hfjm9\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.494804 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.490753    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-dshm\") pod \"distinct-w5-278ffa-kserve-5995c5f648-hfjm9\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.494804 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.490793    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-tls-certs\") pod \"distinct-w5-278ffa-kserve-5995c5f648-hfjm9\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.494804 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.490866    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-model-cache\") pod \"distinct-w5-278ffa-kserve-5995c5f648-hfjm9\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.494804 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.491363    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-tmp-dir\") pod \"distinct-w5-278ffa-kserve-5995c5f648-hfjm9\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.494804 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.491575    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-home\") pod \"distinct-w5-278ffa-kserve-5995c5f648-hfjm9\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.494804 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.492694    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-model-cache\") pod \"distinct-w5-278ffa-kserve-5995c5f648-hfjm9\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.496471 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.496426    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-tls-certs\") pod \"distinct-w5-278ffa-kserve-5995c5f648-hfjm9\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.496868 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.496840    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-dshm\") pod \"distinct-w5-278ffa-kserve-5995c5f648-hfjm9\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.508716 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.508688    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbhd8\" (UniqueName: \"kubernetes.io/projected/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-kube-api-access-wbhd8\") pod \"distinct-w5-278ffa-kserve-5995c5f648-hfjm9\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.558671 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.558643    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:10.725641 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.725612    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"]
Oct 30 19:52:10.728124 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:52:10.728093    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d97d5bd_2d70_4d23_bc5f_bbcc9abf45d7.slice/crio-821ac2d65e3f9cf8ffed35bb99876e43b1ccca20fbdafc72c9c641627f4cc558 WatchSource:0}: Error finding container 821ac2d65e3f9cf8ffed35bb99876e43b1ccca20fbdafc72c9c641627f4cc558: Status 404 returned error can't find the container with id 821ac2d65e3f9cf8ffed35bb99876e43b1ccca20fbdafc72c9c641627f4cc558
Oct 30 19:52:10.730571 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.730536    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:10.730642 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.730603    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:10.730642 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.730630    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:10.819463 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.819420    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m" event={"ID":"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7","Type":"ContainerStarted","Data":"4f18ddb603efa9d64eb0330288224bb0aafd48a286ac36a0cb63e991892981c5"}
Oct 30 19:52:10.819625 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.819476    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m" event={"ID":"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7","Type":"ContainerStarted","Data":"821ac2d65e3f9cf8ffed35bb99876e43b1ccca20fbdafc72c9c641627f4cc558"}
Oct 30 19:52:10.844403 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.844369    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"]
Oct 30 19:52:10.849518 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:10.849478    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m" podStartSLOduration=0.849464437 podStartE2EDuration="849.464437ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:10.848366032 +0000 UTC m=+1285.432635648" watchObservedRunningTime="2026-10-30 19:52:10.849464437 +0000 UTC m=+1285.433734053"
Oct 30 19:52:10.850194 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:52:10.849491    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76e5be67_6a55_4e8f_94dc_1a64e11d7f65.slice/crio-b928f0361d84576174901b7df7ac64f352b6c8b12c565737627e4c5fced32c75 WatchSource:0}: Error finding container b928f0361d84576174901b7df7ac64f352b6c8b12c565737627e4c5fced32c75: Status 404 returned error can't find the container with id b928f0361d84576174901b7df7ac64f352b6c8b12c565737627e4c5fced32c75
Oct 30 19:52:11.389180 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:11.389147    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:11.390568 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:11.390537    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 19:52:11.390706 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:11.390592    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m" podUID="4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 19:52:11.832151 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:11.832115    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" event={"ID":"76e5be67-6a55-4e8f-94dc-1a64e11d7f65","Type":"ContainerStarted","Data":"0d717eb226e8b5c73e64bc593680dcd58c2638f38ebf3798f0ef13e4d07aab0a"}
Oct 30 19:52:11.832700 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:11.832676    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" event={"ID":"76e5be67-6a55-4e8f-94dc-1a64e11d7f65","Type":"ContainerStarted","Data":"b928f0361d84576174901b7df7ac64f352b6c8b12c565737627e4c5fced32c75"}
Oct 30 19:52:11.859064 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:11.859007    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podStartSLOduration=1.8589949049999999 podStartE2EDuration="1.858994905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:11.858399773 +0000 UTC m=+1286.442669392" watchObservedRunningTime="2026-10-30 19:52:11.858994905 +0000 UTC m=+1286.443264520"
Oct 30 19:52:12.389195 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:12.389164    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 19:52:12.389412 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:12.389216    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m" podUID="4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 19:52:12.559763 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:12.559729    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:13.388788 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:13.388744    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 19:52:13.389315 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:13.388846    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m" podUID="4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 19:52:14.388677 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:14.388633    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 19:52:14.388829 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:14.388692    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m" podUID="4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 19:52:15.388577 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:15.388542    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 19:52:15.388825 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:15.388801    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m" podUID="4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 19:52:16.389760 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:16.389708    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 19:52:16.390256 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:16.389780    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m" podUID="4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 19:52:17.694942 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:17.694907    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:17.696175 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:17.696152    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:17.697218 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:17.697197    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 19:52:19.494055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:19.493855    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-h68ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:19.494055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:19.493933    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls" podUID="e6ba43a0-ddfc-459d-8e82-1c2e197557a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:20.559332 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:20.559293    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:20.571496 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:20.571475    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:20.902006 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:20.901940    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:25.374317 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.374276    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"]
Oct 30 19:52:25.378384 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.378361    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.408966 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.408937    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ced832-openshift-default-dockercfg-4thzg\""
Oct 30 19:52:25.409222 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.409175    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"]
Oct 30 19:52:25.458913 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.458882    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.458913 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.458915    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4slp\" (UniqueName: \"kubernetes.io/projected/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-kube-api-access-w4slp\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.459132 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.458949    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.459132 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.459085    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.459132 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.459117    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.459132 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.459132    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.459270 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.459183    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.459308 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.459270    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.459343 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.459316    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.560435 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.560402    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.560683 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.560464    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.560683 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.560497    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.560683 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.560518    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.560683 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.560544    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.560683 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.560593    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.560683 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.560663    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.560991 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.560772    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.560991 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.560799    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4slp\" (UniqueName: \"kubernetes.io/projected/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-kube-api-access-w4slp\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.560991 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.560844    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.560991 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.560954    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.561337 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.561308    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.561473 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.561345    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.561473 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.561410    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.562891 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.562865    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.563140 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.563126    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.578761 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.578732    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.579720 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.579699    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4slp\" (UniqueName: \"kubernetes.io/projected/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-kube-api-access-w4slp\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.666356 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.666282    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"]
Oct 30 19:52:25.670957 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.670940    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.690685 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.690658    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a1d885-openshift-default-dockercfg-stn6c\""
Oct 30 19:52:25.692842 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.692825    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:25.700465 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.700439    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"]
Oct 30 19:52:25.766355 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.765596    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.766355 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.765651    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.766355 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.765687    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.766355 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.765727    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/795803a2-94c7-4773-b51c-5f84a33a28f7-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.766355 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.765793    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.766355 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.765825    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qmhx\" (UniqueName: \"kubernetes.io/projected/795803a2-94c7-4773-b51c-5f84a33a28f7-kube-api-access-5qmhx\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.766355 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.765887    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.766355 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.765984    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.766355 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.766022    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.779610 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.779467    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"]
Oct 30 19:52:25.787320 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.787293    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.813209 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.813184    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-278ffa-openshift-default-dockercfg-9bwm7\""
Oct 30 19:52:25.828094 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.828066    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"]
Oct 30 19:52:25.866651 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.866623    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.866864 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.866835    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.866942 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.866867    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2a0744-9e32-4896-914e-dfb9173923e7-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.866942 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.866904    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.867076 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.866941    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.867076 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.866971    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.867076 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.867013    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/795803a2-94c7-4773-b51c-5f84a33a28f7-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.867229 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.867073    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.867229 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.867106    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qmhx\" (UniqueName: \"kubernetes.io/projected/795803a2-94c7-4773-b51c-5f84a33a28f7-kube-api-access-5qmhx\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.867229 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.867130    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.867229 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.867199    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.870101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.867251    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.870101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.868950    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcj94\" (UniqueName: \"kubernetes.io/projected/6c2a0744-9e32-4896-914e-dfb9173923e7-kube-api-access-pcj94\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.870101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.869076    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.870101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.869107    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.870101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.869134    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.870101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.869171    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.870101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.869240    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.875207 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.874806    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.875819 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.875787    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.877057 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.876867    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.877170 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.877060    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.877253 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.877216    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.877446 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.877421    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/795803a2-94c7-4773-b51c-5f84a33a28f7-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.879247 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.879225    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.899900 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.899878    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qmhx\" (UniqueName: \"kubernetes.io/projected/795803a2-94c7-4773-b51c-5f84a33a28f7-kube-api-access-5qmhx\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.902625 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.902599    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.916563 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.916505    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"]
Oct 30 19:52:25.921307 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.921272    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:25.930093 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.930076    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5f933e-openshift-default-dockercfg-dspvq\""
Oct 30 19:52:25.935512 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.935489    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"]
Oct 30 19:52:25.970988 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.970958    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.971171 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.971018    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.971171 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.971061    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcj94\" (UniqueName: \"kubernetes.io/projected/6c2a0744-9e32-4896-914e-dfb9173923e7-kube-api-access-pcj94\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.971171 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.971093    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.971171 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.971109    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.971171 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.971129    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.971469 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.971177    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.971469 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.971201    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2a0744-9e32-4896-914e-dfb9173923e7-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.971469 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.971246    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.971858 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.971819    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.971964 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.971863    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.971964 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.971929    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.972074 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.972005    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2a0744-9e32-4896-914e-dfb9173923e7-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.972205 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.972184    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.973597 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.973577    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.973773 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.973732    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:25.985102 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.984859    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:25.986436 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.986415    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"]
Oct 30 19:52:25.988461 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:52:25.988440    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe054e1_8ae6_4066_a04c_94a49bddb6a2.slice/crio-4ae6f168557f60f9616dbb3887e89afa7d48a1e830ac38804750ee832eb90334 WatchSource:0}: Error finding container 4ae6f168557f60f9616dbb3887e89afa7d48a1e830ac38804750ee832eb90334: Status 404 returned error can't find the container with id 4ae6f168557f60f9616dbb3887e89afa7d48a1e830ac38804750ee832eb90334
Oct 30 19:52:26.003223 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.995788    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:26.003223 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.995778    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:26.003223 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.995915    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:26.003223 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.996021    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:26.003223 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:25.998634    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcj94\" (UniqueName: \"kubernetes.io/projected/6c2a0744-9e32-4896-914e-dfb9173923e7-kube-api-access-pcj94\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:26.074796 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.072442    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.074796 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.072493    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.074796 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.072527    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a746e113-69a0-4d13-acf1-bb28ba1769b7-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.074796 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.072566    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.074796 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.072619    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.074796 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.072684    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f9ks\" (UniqueName: \"kubernetes.io/projected/a746e113-69a0-4d13-acf1-bb28ba1769b7-kube-api-access-4f9ks\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.074796 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.072801    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.074796 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.072858    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.074796 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.072923    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.101820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.101791    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:26.156066 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.152663    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"]
Oct 30 19:52:26.157817 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.157796    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.180790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.169310    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8a8207-openshift-default-dockercfg-shmd6\""
Oct 30 19:52:26.180790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.173641    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a746e113-69a0-4d13-acf1-bb28ba1769b7-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.180790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.173688    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.180790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.173733    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.180790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.173761    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f9ks\" (UniqueName: \"kubernetes.io/projected/a746e113-69a0-4d13-acf1-bb28ba1769b7-kube-api-access-4f9ks\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.180790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.173845    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.180790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.173957    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.180790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.174018    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.180790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.175185    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.180790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.175223    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.180790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.175573    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a746e113-69a0-4d13-acf1-bb28ba1769b7-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.180790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.177482    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.180790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.178763    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.180790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.180019    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"]
Oct 30 19:52:26.184167 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.183563    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.189058 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.185581    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.189058 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.185775    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.189058 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.186002    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.192782 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.192754    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f9ks\" (UniqueName: \"kubernetes.io/projected/a746e113-69a0-4d13-acf1-bb28ba1769b7-kube-api-access-4f9ks\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.194497 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.194475    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.234887 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.234850    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:26.276492 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.276457    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.276641 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.276509    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.276641 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.276575    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.276641 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.276611    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.276744 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.276691    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g5bm\" (UniqueName: \"kubernetes.io/projected/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-kube-api-access-5g5bm\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.276744 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.276711    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.276807 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.276765    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.276842 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.276833    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.276877 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.276851    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.301464 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.301372    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"]
Oct 30 19:52:26.310079 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:52:26.310013    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795803a2_94c7_4773_b51c_5f84a33a28f7.slice/crio-a58913f59822dfbfce038e59b2d89beae38698f5cca7951d43570c9d6c147d3a WatchSource:0}: Error finding container a58913f59822dfbfce038e59b2d89beae38698f5cca7951d43570c9d6c147d3a: Status 404 returned error can't find the container with id a58913f59822dfbfce038e59b2d89beae38698f5cca7951d43570c9d6c147d3a
Oct 30 19:52:26.313086 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.313054    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:26.313176 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.313117    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:26.313176 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.313148    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:26.378522 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.378475    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g5bm\" (UniqueName: \"kubernetes.io/projected/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-kube-api-access-5g5bm\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.378958 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.378536    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.378958 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.378568    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.378958 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.378668    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.378958 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.378699    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.378958 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.378802    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.379291 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.379155    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.379291 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.379192    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.379397 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.379287    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.379397 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.379338    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.379397 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.379368    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.379565 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.379543    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.379740 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.379704    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.379971 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.379948    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.381647 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.381629    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.381817 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.381798    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.393467 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.393430    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.393999 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.393976    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g5bm\" (UniqueName: \"kubernetes.io/projected/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-kube-api-access-5g5bm\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.404563 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.404538    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"]
Oct 30 19:52:26.411667 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:52:26.411641    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2a0744_9e32_4896_914e_dfb9173923e7.slice/crio-95c635a31a768de95686f19c6b6c119980b0c6d4ced212b72d331c629ea85bf2 WatchSource:0}: Error finding container 95c635a31a768de95686f19c6b6c119980b0c6d4ced212b72d331c629ea85bf2: Status 404 returned error can't find the container with id 95c635a31a768de95686f19c6b6c119980b0c6d4ced212b72d331c629ea85bf2
Oct 30 19:52:26.414156 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.414126    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:26.414226 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.414190    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:26.414226 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.414219    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:26.489634 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.489596    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:26.528845 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.528802    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"]
Oct 30 19:52:26.535555 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:52:26.535523    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda746e113_69a0_4d13_acf1_bb28ba1769b7.slice/crio-3252c229407e402bed01e0a9ef043ec81e26bf01c37e0f1dc4ca1648c6ccc225 WatchSource:0}: Error finding container 3252c229407e402bed01e0a9ef043ec81e26bf01c37e0f1dc4ca1648c6ccc225: Status 404 returned error can't find the container with id 3252c229407e402bed01e0a9ef043ec81e26bf01c37e0f1dc4ca1648c6ccc225
Oct 30 19:52:26.541549 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.541506    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:26.541672 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.541598    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:26.541672 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.541641    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:26.851699 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.851656    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"]
Oct 30 19:52:26.875219 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.874763    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:26.875219 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.874845    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:26.875219 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.874884    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:26.960333 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.960237    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2" event={"ID":"795803a2-94c7-4773-b51c-5f84a33a28f7","Type":"ContainerStarted","Data":"20f09078c860f497fb00f4f3a22f1a58a3137914a04180b14b7641061e9752a7"}
Oct 30 19:52:26.960595 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.960556    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2" event={"ID":"795803a2-94c7-4773-b51c-5f84a33a28f7","Type":"ContainerStarted","Data":"a58913f59822dfbfce038e59b2d89beae38698f5cca7951d43570c9d6c147d3a"}
Oct 30 19:52:26.964091 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.964067    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq" event={"ID":"8fe054e1-8ae6-4066-a04c-94a49bddb6a2","Type":"ContainerStarted","Data":"29b95f6a3170c5c292a2f34b2c8a15443a74880f67aae138e30347ceab11111d"}
Oct 30 19:52:26.964268 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.964251    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq" event={"ID":"8fe054e1-8ae6-4066-a04c-94a49bddb6a2","Type":"ContainerStarted","Data":"4ae6f168557f60f9616dbb3887e89afa7d48a1e830ac38804750ee832eb90334"}
Oct 30 19:52:26.967264 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.967196    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" event={"ID":"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad","Type":"ContainerStarted","Data":"d2d2045f1da198eaf5582c0e85545da77a1b3918d7ce6334a2bca9f407e0523e"}
Oct 30 19:52:26.981323 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.981264    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb" event={"ID":"6c2a0744-9e32-4896-914e-dfb9173923e7","Type":"ContainerStarted","Data":"da9f1f5f7532d452ee472e50791f1e02e3637f6af2292d06fedbd9a3834fbb31"}
Oct 30 19:52:26.981571 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.981542    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb" event={"ID":"6c2a0744-9e32-4896-914e-dfb9173923e7","Type":"ContainerStarted","Data":"95c635a31a768de95686f19c6b6c119980b0c6d4ced212b72d331c629ea85bf2"}
Oct 30 19:52:26.989101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.987542    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" event={"ID":"a746e113-69a0-4d13-acf1-bb28ba1769b7","Type":"ContainerStarted","Data":"d80ce52ae2d7be751312ece6e8ef83d805957bb8d83aa34af42f60928ac70501"}
Oct 30 19:52:26.989101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.987593    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" event={"ID":"a746e113-69a0-4d13-acf1-bb28ba1769b7","Type":"ContainerStarted","Data":"3252c229407e402bed01e0a9ef043ec81e26bf01c37e0f1dc4ca1648c6ccc225"}
Oct 30 19:52:26.989101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.988293    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:26.996073 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:26.995994    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2" podStartSLOduration=1.9959726770000001 podStartE2EDuration="1.995972677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:26.994217775 +0000 UTC m=+1301.578487393" watchObservedRunningTime="2026-10-30 19:52:26.995972677 +0000 UTC m=+1301.580242307"
Oct 30 19:52:27.049424 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:27.048961    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" podStartSLOduration=2.048939189 podStartE2EDuration="2.048939189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:27.022292956 +0000 UTC m=+1301.606562585" watchObservedRunningTime="2026-10-30 19:52:27.048939189 +0000 UTC m=+1301.633208804"
Oct 30 19:52:27.078710 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:27.074000    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq" podStartSLOduration=2.073981514 podStartE2EDuration="2.073981514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:27.072872868 +0000 UTC m=+1301.657142487" watchObservedRunningTime="2026-10-30 19:52:27.073981514 +0000 UTC m=+1301.658251132"
Oct 30 19:52:27.106430 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:27.106357    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:27.158266 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:27.145134    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:52:27.158266 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:27.145190    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb" podUID="6c2a0744-9e32-4896-914e-dfb9173923e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 19:52:27.237057 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:27.236764    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:27.237057 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:27.236892    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:52:27.237057 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:27.236935    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 19:52:27.697620 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:27.694381    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:27.710083 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:27.710013    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:52:27.710251 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:27.710101    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq" podUID="8fe054e1-8ae6-4066-a04c-94a49bddb6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 19:52:27.992053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:27.990658    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:52:27.992053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:27.990730    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2" podUID="795803a2-94c7-4773-b51c-5f84a33a28f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 19:52:28.026432 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:28.026387    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" event={"ID":"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad","Type":"ContainerStarted","Data":"55539d73b315aa3a84ad5975871bbfde61277cd7de8088e65702be91617d8a57"}
Oct 30 19:52:28.078227 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:28.078166    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb" podStartSLOduration=3.078144363 podStartE2EDuration="3.078144363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:27.139706742 +0000 UTC m=+1301.723976359" watchObservedRunningTime="2026-10-30 19:52:28.078144363 +0000 UTC m=+1302.662413981"
Oct 30 19:52:28.104395 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:28.104360    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:52:28.104658 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:28.104633    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb" podUID="6c2a0744-9e32-4896-914e-dfb9173923e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 19:52:28.235482 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:28.235443    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:52:28.235787 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:28.235761    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 19:52:28.494111 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:28.492665    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:28.494111 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:28.492786    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:52:28.494111 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:28.492830    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" podUID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 19:52:28.744094 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:28.743396    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:52:28.744094 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:28.743470    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq" podUID="8fe054e1-8ae6-4066-a04c-94a49bddb6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 19:52:28.986524 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:28.986417    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:52:28.986773 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:28.986749    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2" podUID="795803a2-94c7-4773-b51c-5f84a33a28f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 19:52:29.103705 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:29.103668    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:52:29.103971 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:29.103939    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb" podUID="6c2a0744-9e32-4896-914e-dfb9173923e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 19:52:29.237101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:29.236386    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:52:29.237101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:29.236440    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 19:52:29.494053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:29.490371    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:52:29.494053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:29.490427    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" podUID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 19:52:29.694053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:29.693513    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:52:29.694053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:29.693603    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq" podUID="8fe054e1-8ae6-4066-a04c-94a49bddb6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 19:52:29.989485 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:29.989448    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:52:29.990077 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:29.990042    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2" podUID="795803a2-94c7-4773-b51c-5f84a33a28f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 19:52:30.106732 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:30.106698    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:52:30.106995 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:30.106966    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb" podUID="6c2a0744-9e32-4896-914e-dfb9173923e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 19:52:30.235483 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:30.235445    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:52:30.235780 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:30.235731    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 19:52:30.377962 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:30.377823    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:30.393756 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:30.393680    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:30.504053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:30.498322    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:52:30.504053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:30.498385    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" podUID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 19:52:30.694502 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:30.694405    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:52:30.694743 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:30.694713    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq" podUID="8fe054e1-8ae6-4066-a04c-94a49bddb6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 19:52:30.988630 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:30.988600    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:52:30.989065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:30.988933    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2" podUID="795803a2-94c7-4773-b51c-5f84a33a28f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 19:52:31.102739 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:31.102692    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:52:31.103335 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:31.103291    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb" podUID="6c2a0744-9e32-4896-914e-dfb9173923e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 19:52:31.264105 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:31.241005    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:52:31.347051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:31.346905    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 19:52:31.347419 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:31.345182    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:31.498088 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:31.498048    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:52:31.502362 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:31.502330    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" podUID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 19:52:31.565254 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:31.565157    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:31.697484 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:31.697446    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:52:31.697764 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:31.697726    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq" podUID="8fe054e1-8ae6-4066-a04c-94a49bddb6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 19:52:31.803158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:31.803114    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:31.896162 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:31.896042    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:31.987460 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:31.987421    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:52:31.987753 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:31.987728    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2" podUID="795803a2-94c7-4773-b51c-5f84a33a28f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 19:52:32.107104 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:32.107073    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:52:32.107674 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:32.107645    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb" podUID="6c2a0744-9e32-4896-914e-dfb9173923e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 19:52:32.236278 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:32.236193    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:52:32.236535 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:32.236505    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 19:52:32.368056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:32.365153    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:32.413182 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:32.413129    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:52:32.497170 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:32.497135    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:52:32.498220 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:32.498175    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" podUID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 19:52:32.701877 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:32.701828    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:52:32.702088 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:32.701900    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq" podUID="8fe054e1-8ae6-4066-a04c-94a49bddb6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 19:52:32.810104 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:32.807117    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:52:32.810104 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:32.807767    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:52:32.837256 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:32.837192    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" podStartSLOduration=6.837172056 podStartE2EDuration="6.837172056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:28.080061151 +0000 UTC m=+1302.664330767" watchObservedRunningTime="2026-10-30 19:52:32.837172056 +0000 UTC m=+1307.421441675"
Oct 30 19:52:32.988398 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:32.988360    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:52:32.988640 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:32.988614    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2" podUID="795803a2-94c7-4773-b51c-5f84a33a28f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 19:52:33.116428 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:33.116347    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:52:33.117342 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:33.117311    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb" podUID="6c2a0744-9e32-4896-914e-dfb9173923e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 19:52:33.243056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:33.235444    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:52:33.243056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:33.235501    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 19:52:33.316625 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:33.316182    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:33.377228 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:33.377097    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:33.378463 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:33.378443    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:52:33.498357 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:33.498300    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:33.499335 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:33.499302    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:52:33.506075 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:33.499504    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" podUID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 19:52:33.709831 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:33.709737    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:52:33.710136 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:33.710091    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq" podUID="8fe054e1-8ae6-4066-a04c-94a49bddb6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 19:52:33.891595 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:33.866684    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:33.987378 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:33.986165    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:52:33.987378 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:33.986226    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2" podUID="795803a2-94c7-4773-b51c-5f84a33a28f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 19:52:34.121496 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:34.121462    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:52:34.121970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:34.121520    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb" podUID="6c2a0744-9e32-4896-914e-dfb9173923e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 19:52:34.242083 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:34.241141    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:52:34.242083 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:34.241205    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 19:52:34.320062 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:34.319117    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:52:34.320062 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:34.319744    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:52:34.379159 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:34.379115    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:34.391412 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:34.391366    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:34.508113 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:34.493415    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:52:34.508686 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:34.508658    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" podUID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 19:52:34.509297 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:34.508580    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:34.702067 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:34.699414    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:52:34.702067 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:34.699476    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq" podUID="8fe054e1-8ae6-4066-a04c-94a49bddb6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 19:52:34.712193 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:34.711562    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:52:34.889215 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:34.889123    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:34.942203 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:34.942153    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:35.007537 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.007500    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:52:35.007791 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.007759    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2" podUID="795803a2-94c7-4773-b51c-5f84a33a28f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 19:52:35.129055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.103581    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:52:35.129771 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.129723    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb" podUID="6c2a0744-9e32-4896-914e-dfb9173923e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 19:52:35.281055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.255688    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:52:35.282377 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.282341    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 19:52:35.383054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.382287    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:35.406180 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.406129    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:35.407924 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.407900    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:52:35.650863 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.646544    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:52:35.721207 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.719842    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:52:35.721207 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.719899    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" podUID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 19:52:35.721207 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.720563    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:52:35.721207 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.720600    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq" podUID="8fe054e1-8ae6-4066-a04c-94a49bddb6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 19:52:35.847055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.821721    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:52:35.893054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.892676    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:52:35.941103 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.940236    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:35.970636 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.970578    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:52:35.971528 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:35.971506    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:36.005571 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:36.005537    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:52:36.005815 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:36.005792    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2" podUID="795803a2-94c7-4773-b51c-5f84a33a28f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 19:52:36.028683 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:36.028647    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:52:36.103961 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:36.103930    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:52:36.247060 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:36.245107    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb" podUID="6c2a0744-9e32-4896-914e-dfb9173923e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 19:52:36.247060 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:36.133878    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:52:36.247060 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:36.242620    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:52:36.247060 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:36.246328    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 19:52:36.397790 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:36.397735    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:36.490464 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:36.490419    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:52:36.490702 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:36.490680    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" podUID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 19:52:36.499491 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:36.499427    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:52:36.500819 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:36.500800    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:52:36.713050 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:36.709956    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:52:36.713050 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:36.710015    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq" podUID="8fe054e1-8ae6-4066-a04c-94a49bddb6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 19:52:37.117059 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:37.116485    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:52:37.117059 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:37.116562    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb" podUID="6c2a0744-9e32-4896-914e-dfb9173923e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 19:52:37.238049 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:37.237482    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:52:37.238049 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:37.237555    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 19:52:37.493476 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:37.493439    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:52:37.494071 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:37.494011    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" podUID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 19:52:38.031177 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:38.031135    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:38.031448 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:38.031413    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2" podUID="795803a2-94c7-4773-b51c-5f84a33a28f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:38.099053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:38.078319    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:38.099053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:38.079381    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:38.099053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:38.091344    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 19:52:38.385387 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:38.385313    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:38.403590 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:38.401442    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:38.480795 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:38.480757    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 19:52:38.490791 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:38.490766    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:52:38.491039 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:38.491005    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" podUID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 19:52:38.514584 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:38.514564    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:38.519764 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:38.519734    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:38.534446 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:38.534419    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 19:52:39.285766 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:39.285211    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:39.285766 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:39.285303    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:39.322062 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:39.321742    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:39.322796 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:39.322773    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:39.323602 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:39.323581    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 19:52:39.680064 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:52:39.679972    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:52:39.963658 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:39.960780    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"]
Oct 30 19:52:39.975491 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:39.975465    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:39.989995 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:39.985816    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d5e3e5-openshift-default-dockercfg-pw4z9\""
Oct 30 19:52:39.997918 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:39.997892    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"]
Oct 30 19:52:40.063051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.059119    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.063051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.059181    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/084af46f-59a7-4518-8938-2b4e09979c5f-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.063051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.059220    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.063051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.059296    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.063051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.059351    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084af46f-59a7-4518-8938-2b4e09979c5f-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.063051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.059408    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.063051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.059447    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbgx6\" (UniqueName: \"kubernetes.io/projected/084af46f-59a7-4518-8938-2b4e09979c5f-kube-api-access-lbgx6\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.063051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.059509    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.063051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.059570    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084af46f-59a7-4518-8938-2b4e09979c5f-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.133795 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.133766    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:40.134106 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.134082    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:40.134928 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.134897    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 19:52:40.161096 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.161065    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbgx6\" (UniqueName: \"kubernetes.io/projected/084af46f-59a7-4518-8938-2b4e09979c5f-kube-api-access-lbgx6\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.161319 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.161138    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.161405 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.161370    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084af46f-59a7-4518-8938-2b4e09979c5f-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.161662 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.161454    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.161662 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.161486    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/084af46f-59a7-4518-8938-2b4e09979c5f-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.161662 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.161513    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.161662 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.161590    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.161662 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.161657    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084af46f-59a7-4518-8938-2b4e09979c5f-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.161936 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.161761    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.163504 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.162639    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.163504 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.162781    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.163504 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.163061    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.164226 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.164202    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084af46f-59a7-4518-8938-2b4e09979c5f-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.164383 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.164328    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.165569 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.165547    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.165999 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.165977    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084af46f-59a7-4518-8938-2b4e09979c5f-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.186178 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.186157    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbgx6\" (UniqueName: \"kubernetes.io/projected/084af46f-59a7-4518-8938-2b4e09979c5f-kube-api-access-lbgx6\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.186717 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.186692    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/084af46f-59a7-4518-8938-2b4e09979c5f-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.313185 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.313149    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:40.750103 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.749824    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"]
Oct 30 19:52:40.752606 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:52:40.752579    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084af46f_59a7_4518_8938_2b4e09979c5f.slice/crio-425b7395aee8b8049cb9e51526c5e95b111203e8ade23f8de908668a7cf3a5b5 WatchSource:0}: Error finding container 425b7395aee8b8049cb9e51526c5e95b111203e8ade23f8de908668a7cf3a5b5: Status 404 returned error can't find the container with id 425b7395aee8b8049cb9e51526c5e95b111203e8ade23f8de908668a7cf3a5b5
Oct 30 19:52:40.755089 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.755060    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:40.755146 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.755123    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:40.755178 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:40.755153    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:52:41.409467 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:41.409429    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc" event={"ID":"084af46f-59a7-4518-8938-2b4e09979c5f","Type":"ContainerStarted","Data":"3ce5c6d61ebeb949b8567f999d0df77009af293288151ad4b7fa388da8f8cb91"}
Oct 30 19:52:41.409798 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:41.409496    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc" event={"ID":"084af46f-59a7-4518-8938-2b4e09979c5f","Type":"ContainerStarted","Data":"425b7395aee8b8049cb9e51526c5e95b111203e8ade23f8de908668a7cf3a5b5"}
Oct 30 19:52:41.440348 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:41.440285    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc" podStartSLOduration=2.440267355 podStartE2EDuration="2.440267355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:41.438730761 +0000 UTC m=+1316.023000376" watchObservedRunningTime="2026-10-30 19:52:41.440267355 +0000 UTC m=+1316.024536969"
Oct 30 19:52:42.327053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:42.326374    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:42.329576 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:42.329533    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:52:42.329863 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:42.329812    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc" podUID="084af46f-59a7-4518-8938-2b4e09979c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 19:52:43.321053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:43.314435    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:52:43.321053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:43.314507    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc" podUID="084af46f-59a7-4518-8938-2b4e09979c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 19:52:44.324461 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:44.324426    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:52:44.325080 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:44.325024    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc" podUID="084af46f-59a7-4518-8938-2b4e09979c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 19:52:45.316482 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:45.316445    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:52:45.317628 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:45.317595    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc" podUID="084af46f-59a7-4518-8938-2b4e09979c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 19:52:46.317516 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:46.317476    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:52:46.318063 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:46.317553    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc" podUID="084af46f-59a7-4518-8938-2b4e09979c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 19:52:47.314125 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:47.314088    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:52:47.314305 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:47.314160    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc" podUID="084af46f-59a7-4518-8938-2b4e09979c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 19:52:48.738009 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:48.737977    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:48.739579 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:48.739553    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:48.742565 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:48.740938    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 19:52:56.302741 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.302706    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"]
Oct 30 19:52:56.309065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.309007    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.316244 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.316211    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-model-cache\") pod \"distinct-w1-a1d885-kserve-579f467465-kgvrk\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.316463 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.316258    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-tmp-dir\") pod \"distinct-w1-a1d885-kserve-579f467465-kgvrk\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.316463 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.316438    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-tls-certs\") pod \"distinct-w1-a1d885-kserve-579f467465-kgvrk\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.316583 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.316495    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sd2l\" (UniqueName: \"kubernetes.io/projected/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-kube-api-access-6sd2l\") pod \"distinct-w1-a1d885-kserve-579f467465-kgvrk\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.316639 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.316610    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-home\") pod \"distinct-w1-a1d885-kserve-579f467465-kgvrk\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.316695 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.316649    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-dshm\") pod \"distinct-w1-a1d885-kserve-579f467465-kgvrk\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.328240 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.328211    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"distinct-w1-a1d885-kserve-self-signed-certs\""
Oct 30 19:52:56.328615 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.328586    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"]
Oct 30 19:52:56.417528 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.417496    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-dshm\") pod \"distinct-w1-a1d885-kserve-579f467465-kgvrk\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.417728 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.417555    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-model-cache\") pod \"distinct-w1-a1d885-kserve-579f467465-kgvrk\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.417728 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.417576    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-tmp-dir\") pod \"distinct-w1-a1d885-kserve-579f467465-kgvrk\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.417728 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.417631    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-tls-certs\") pod \"distinct-w1-a1d885-kserve-579f467465-kgvrk\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.417728 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.417661    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sd2l\" (UniqueName: \"kubernetes.io/projected/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-kube-api-access-6sd2l\") pod \"distinct-w1-a1d885-kserve-579f467465-kgvrk\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.417728 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.417725    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-home\") pod \"distinct-w1-a1d885-kserve-579f467465-kgvrk\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.418157 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.418129    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-home\") pod \"distinct-w1-a1d885-kserve-579f467465-kgvrk\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.418493 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.418464    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-tmp-dir\") pod \"distinct-w1-a1d885-kserve-579f467465-kgvrk\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.418742 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.418721    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-model-cache\") pod \"distinct-w1-a1d885-kserve-579f467465-kgvrk\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.420987 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.420958    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-tls-certs\") pod \"distinct-w1-a1d885-kserve-579f467465-kgvrk\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.421182 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.421160    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-dshm\") pod \"distinct-w1-a1d885-kserve-579f467465-kgvrk\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.449051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.434360    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sd2l\" (UniqueName: \"kubernetes.io/projected/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-kube-api-access-6sd2l\") pod \"distinct-w1-a1d885-kserve-579f467465-kgvrk\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:56.629115 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:56.628934    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:52:57.064108 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:57.064074    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"]
Oct 30 19:52:57.068390 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:52:57.068365    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a886ba9_fe5e_4739_bb26_7bdb235e6e64.slice/crio-74d8e1fad8227b8d3db82adfe859a613ee335428edab0d9b972201a7eab52d76 WatchSource:0}: Error finding container 74d8e1fad8227b8d3db82adfe859a613ee335428edab0d9b972201a7eab52d76: Status 404 returned error can't find the container with id 74d8e1fad8227b8d3db82adfe859a613ee335428edab0d9b972201a7eab52d76
Oct 30 19:52:57.558643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:57.558612    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" event={"ID":"6a886ba9-fe5e-4739-bb26-7bdb235e6e64","Type":"ContainerStarted","Data":"4299d2ed3393872930a03174f345fe1512fa6c322d77e02d128d40ed01a927f2"}
Oct 30 19:52:57.558643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:57.558649    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" event={"ID":"6a886ba9-fe5e-4739-bb26-7bdb235e6e64","Type":"ContainerStarted","Data":"74d8e1fad8227b8d3db82adfe859a613ee335428edab0d9b972201a7eab52d76"}
Oct 30 19:52:57.588266 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:57.588219    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podStartSLOduration=1.588205244 podStartE2EDuration="1.588205244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:57.585575844 +0000 UTC m=+1332.169845460" watchObservedRunningTime="2026-10-30 19:52:57.588205244 +0000 UTC m=+1332.172474860"
Oct 30 19:52:57.630192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:52:57.630165    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:53:06.629587 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:06.629552    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:53:06.641818 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:06.641794    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:53:07.829055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:07.828696    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:53:09.661821 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:53:09.661780    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:53:10.225660 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.220096    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"]
Oct 30 19:53:10.225660 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.225329    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.256971 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.252190    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"]
Oct 30 19:53:10.384502 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.384469    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.384660 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.384540    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wp2h\" (UniqueName: \"kubernetes.io/projected/75115e13-a71f-49ff-8bbc-6df7eebd4195-kube-api-access-2wp2h\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.384660 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.384603    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.384660 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.384643    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.384764 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.384662    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.384764 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.384718    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.384764 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.384747    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.384901 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.384886    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75115e13-a71f-49ff-8bbc-6df7eebd4195-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.384951 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.384936    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.486567 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.486470    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.486711 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.486577    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.486711 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.486636    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wp2h\" (UniqueName: \"kubernetes.io/projected/75115e13-a71f-49ff-8bbc-6df7eebd4195-kube-api-access-2wp2h\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.486711 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.486674    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.486810 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.486730    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.486877 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.486855    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.486979 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.486957    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.487065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.487002    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.487065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.487017    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.487169 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.487067    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.487169 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.487111    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75115e13-a71f-49ff-8bbc-6df7eebd4195-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.487327 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.487292    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.487410 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.487331    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.487745 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.487726    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75115e13-a71f-49ff-8bbc-6df7eebd4195-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.488912 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.488890    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.489467 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.489446    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.501980 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.501957    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wp2h\" (UniqueName: \"kubernetes.io/projected/75115e13-a71f-49ff-8bbc-6df7eebd4195-kube-api-access-2wp2h\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.505050 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.505015    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.564672 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.564652    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:10.856904 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.856862    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"]
Oct 30 19:53:10.859392 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:53:10.859364    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75115e13_a71f_49ff_8bbc_6df7eebd4195.slice/crio-9a94276aca74646ebc9af617bf3845ba0838ac1f44ad6187199da187b17b915c WatchSource:0}: Error finding container 9a94276aca74646ebc9af617bf3845ba0838ac1f44ad6187199da187b17b915c: Status 404 returned error can't find the container with id 9a94276aca74646ebc9af617bf3845ba0838ac1f44ad6187199da187b17b915c
Oct 30 19:53:10.861757 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.861720    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:10.861867 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.861784    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:10.861867 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:10.861816    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:11.683553 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:11.683521    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm" event={"ID":"75115e13-a71f-49ff-8bbc-6df7eebd4195","Type":"ContainerStarted","Data":"6754b42b0f5681bf1388824942b0e8736f59e6217296b7acbc2c780254e55d07"}
Oct 30 19:53:11.683713 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:11.683573    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm" event={"ID":"75115e13-a71f-49ff-8bbc-6df7eebd4195","Type":"ContainerStarted","Data":"9a94276aca74646ebc9af617bf3845ba0838ac1f44ad6187199da187b17b915c"}
Oct 30 19:53:11.729844 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:11.720893    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm" podStartSLOduration=1.720862082 podStartE2EDuration="1.720862082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:11.714851941 +0000 UTC m=+1346.299121557" watchObservedRunningTime="2026-10-30 19:53:11.720862082 +0000 UTC m=+1346.305131699"
Oct 30 19:53:12.574246 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:12.573272    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:12.574246 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:12.573435    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:53:12.574246 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:12.573510    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm" podUID="75115e13-a71f-49ff-8bbc-6df7eebd4195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 19:53:13.565491 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:13.565458    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:53:13.565816 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:13.565786    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm" podUID="75115e13-a71f-49ff-8bbc-6df7eebd4195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 19:53:14.565962 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:14.565924    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:53:14.566515 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:14.565978    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm" podUID="75115e13-a71f-49ff-8bbc-6df7eebd4195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 19:53:15.565921 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:15.565889    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:53:15.566102 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:15.565947    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm" podUID="75115e13-a71f-49ff-8bbc-6df7eebd4195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 19:53:16.570756 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:16.570541    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:53:16.570756 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:16.570617    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm" podUID="75115e13-a71f-49ff-8bbc-6df7eebd4195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 19:53:16.977592 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:16.977518    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"]
Oct 30 19:53:16.982668 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:16.982645    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:16.987583 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:16.987554    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"openshift-service-ca.crt\""
Oct 30 19:53:16.988837 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:16.988812    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"kube-root-ca.crt\""
Oct 30 19:53:16.988943 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:16.988891    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"distinct-2-w2-d5e3e5-kserve-self-signed-certs\""
Oct 30 19:53:16.988983 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:16.988953    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"default-dockercfg-zwdzq\""
Oct 30 19:53:16.998205 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:16.998185    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"]
Oct 30 19:53:17.096411 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.096374    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-home\") pod \"distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.096615 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.096586    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-tmp-dir\") pod \"distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.096671 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.096618    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-dshm\") pod \"distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.096671 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.096639    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kqqb\" (UniqueName: \"kubernetes.io/projected/3a38c258-0d4e-4b70-afec-45f6025fdcc6-kube-api-access-7kqqb\") pod \"distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.096777 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.096751    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-model-cache\") pod \"distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.096873 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.096856    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a38c258-0d4e-4b70-afec-45f6025fdcc6-tls-certs\") pod \"distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.199056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.198739    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-home\") pod \"distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.199056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.198935    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-tmp-dir\") pod \"distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.199056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.198970    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-dshm\") pod \"distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.199056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.198998    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kqqb\" (UniqueName: \"kubernetes.io/projected/3a38c258-0d4e-4b70-afec-45f6025fdcc6-kube-api-access-7kqqb\") pod \"distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.199056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.199060    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-model-cache\") pod \"distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.199485 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.199098    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a38c258-0d4e-4b70-afec-45f6025fdcc6-tls-certs\") pod \"distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.200307 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.200278    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-home\") pod \"distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.200772 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.200750    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-tmp-dir\") pod \"distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.207049 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.201495    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-model-cache\") pod \"distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.210576 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.210542    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-dshm\") pod \"distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.234126 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.232779    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kqqb\" (UniqueName: \"kubernetes.io/projected/3a38c258-0d4e-4b70-afec-45f6025fdcc6-kube-api-access-7kqqb\") pod \"distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.234126 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.233341    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a38c258-0d4e-4b70-afec-45f6025fdcc6-tls-certs\") pod \"distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.297249 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.297216    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:17.565189 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.565155    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:53:17.565382 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.565208    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm" podUID="75115e13-a71f-49ff-8bbc-6df7eebd4195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 19:53:17.614086 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.614059    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"]
Oct 30 19:53:17.617057 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:53:17.617012    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a38c258_0d4e_4b70_afec_45f6025fdcc6.slice/crio-d531451dfe33f687b6bf57510a8a405ebed678e2cd97f123e942f84817051c57 WatchSource:0}: Error finding container d531451dfe33f687b6bf57510a8a405ebed678e2cd97f123e942f84817051c57: Status 404 returned error can't find the container with id d531451dfe33f687b6bf57510a8a405ebed678e2cd97f123e942f84817051c57
Oct 30 19:53:17.768052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.763578    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" event={"ID":"3a38c258-0d4e-4b70-afec-45f6025fdcc6","Type":"ContainerStarted","Data":"be85bf2e96ba8a10005fce6b6d4288a5aecb57b96861ce1a0054bfa83cfb207c"}
Oct 30 19:53:17.768052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.763636    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" event={"ID":"3a38c258-0d4e-4b70-afec-45f6025fdcc6","Type":"ContainerStarted","Data":"d531451dfe33f687b6bf57510a8a405ebed678e2cd97f123e942f84817051c57"}
Oct 30 19:53:17.806661 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:17.806613    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podStartSLOduration=1.806599635 podStartE2EDuration="1.806599635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:17.805328923 +0000 UTC m=+1352.389598543" watchObservedRunningTime="2026-10-30 19:53:17.806599635 +0000 UTC m=+1352.390869251"
Oct 30 19:53:18.298718 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:18.298680    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:18.569782 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:18.569713    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:18.570020 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:18.570002    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:18.570725 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:18.570705    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 19:53:23.275367 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.275335    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"]
Oct 30 19:53:23.279207 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.279185    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.292540 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.292520    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"default-dockercfg-dsn78\""
Oct 30 19:53:23.307834 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.307809    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"distinct-2-w6-5f933e-kserve-self-signed-certs\""
Oct 30 19:53:23.309162 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.309141    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"openshift-service-ca.crt\""
Oct 30 19:53:23.309301 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.309249    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"kube-root-ca.crt\""
Oct 30 19:53:23.316126 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.316078    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"]
Oct 30 19:53:23.385381 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.385341    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwk5w\" (UniqueName: \"kubernetes.io/projected/920fc884-8d5e-40e5-9e67-550521b12e3e-kube-api-access-kwk5w\") pod \"distinct-2-w6-5f933e-kserve-f9448484c-rlgnb\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.385530 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.385389    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-dshm\") pod \"distinct-2-w6-5f933e-kserve-f9448484c-rlgnb\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.385530 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.385506    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-tmp-dir\") pod \"distinct-2-w6-5f933e-kserve-f9448484c-rlgnb\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.385625 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.385559    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-model-cache\") pod \"distinct-2-w6-5f933e-kserve-f9448484c-rlgnb\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.385813 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.385792    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920fc884-8d5e-40e5-9e67-550521b12e3e-tls-certs\") pod \"distinct-2-w6-5f933e-kserve-f9448484c-rlgnb\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.385915 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.385895    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-home\") pod \"distinct-2-w6-5f933e-kserve-f9448484c-rlgnb\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.503055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.489870    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-home\") pod \"distinct-2-w6-5f933e-kserve-f9448484c-rlgnb\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.503055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.489972    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwk5w\" (UniqueName: \"kubernetes.io/projected/920fc884-8d5e-40e5-9e67-550521b12e3e-kube-api-access-kwk5w\") pod \"distinct-2-w6-5f933e-kserve-f9448484c-rlgnb\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.503055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.489999    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-dshm\") pod \"distinct-2-w6-5f933e-kserve-f9448484c-rlgnb\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.503055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.490055    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-tmp-dir\") pod \"distinct-2-w6-5f933e-kserve-f9448484c-rlgnb\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.503055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.490102    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-model-cache\") pod \"distinct-2-w6-5f933e-kserve-f9448484c-rlgnb\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.503055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.490260    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920fc884-8d5e-40e5-9e67-550521b12e3e-tls-certs\") pod \"distinct-2-w6-5f933e-kserve-f9448484c-rlgnb\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.523064 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.504150    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920fc884-8d5e-40e5-9e67-550521b12e3e-tls-certs\") pod \"distinct-2-w6-5f933e-kserve-f9448484c-rlgnb\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.523064 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.504524    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-home\") pod \"distinct-2-w6-5f933e-kserve-f9448484c-rlgnb\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.523064 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.518641    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-tmp-dir\") pod \"distinct-2-w6-5f933e-kserve-f9448484c-rlgnb\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.523064 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.518960    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-model-cache\") pod \"distinct-2-w6-5f933e-kserve-f9448484c-rlgnb\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.544255 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.544176    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-dshm\") pod \"distinct-2-w6-5f933e-kserve-f9448484c-rlgnb\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.571803 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.571712    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwk5w\" (UniqueName: \"kubernetes.io/projected/920fc884-8d5e-40e5-9e67-550521b12e3e-kube-api-access-kwk5w\") pod \"distinct-2-w6-5f933e-kserve-f9448484c-rlgnb\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.598629 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.598275    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:23.914693 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:23.914673    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"]
Oct 30 19:53:23.916946 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:53:23.916917    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod920fc884_8d5e_40e5_9e67_550521b12e3e.slice/crio-a21f64cf225bb1f46cfbfb541303e1ce958daaeed2efadaf88e8d61b58942aab WatchSource:0}: Error finding container a21f64cf225bb1f46cfbfb541303e1ce958daaeed2efadaf88e8d61b58942aab: Status 404 returned error can't find the container with id a21f64cf225bb1f46cfbfb541303e1ce958daaeed2efadaf88e8d61b58942aab
Oct 30 19:53:24.807756 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:24.807723    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" event={"ID":"920fc884-8d5e-40e5-9e67-550521b12e3e","Type":"ContainerStarted","Data":"960674a42a3f997671a402c0590e536b66f2b0f9077219cd44bfbab437290b2a"}
Oct 30 19:53:24.808187 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:24.807767    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" event={"ID":"920fc884-8d5e-40e5-9e67-550521b12e3e","Type":"ContainerStarted","Data":"a21f64cf225bb1f46cfbfb541303e1ce958daaeed2efadaf88e8d61b58942aab"}
Oct 30 19:53:24.835472 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:24.835408    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podStartSLOduration=1.8353913080000002 podStartE2EDuration="1.835391308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:24.832982549 +0000 UTC m=+1359.417252166" watchObservedRunningTime="2026-10-30 19:53:24.835391308 +0000 UTC m=+1359.419660925"
Oct 30 19:53:25.599498 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:25.599463    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:25.772666 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:25.772617    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"]
Oct 30 19:53:25.776139 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:25.776121    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:25.795925 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:25.795871    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"]
Oct 30 19:53:25.936139 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:25.936053    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:25.936139 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:25.936117    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:25.936609 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:25.936224    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:25.936609 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:25.936283    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:25.936609 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:25.936445    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:25.936609 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:25.936531    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:25.936781 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:25.936688    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:25.936781 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:25.936758    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw9rs\" (UniqueName: \"kubernetes.io/projected/2d3a55c8-98fe-4911-b3c5-476a14bc809a-kube-api-access-gw9rs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:25.936888 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:25.936791    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.038423 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.038388    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.038611 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.038438    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw9rs\" (UniqueName: \"kubernetes.io/projected/2d3a55c8-98fe-4911-b3c5-476a14bc809a-kube-api-access-gw9rs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.038611 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.038466    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.038611 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.038535    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.038611 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.038564    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.038611 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.038601    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.038886 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.038628    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.038886 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.038675    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.038886 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.038724    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.039057 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.038887    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.039057 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.038920    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.039204 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.039183    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.039204 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.039194    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.039306 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.039249    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.041142 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.041114    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.041354 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.041337    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.063300 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.063276    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw9rs\" (UniqueName: \"kubernetes.io/projected/2d3a55c8-98fe-4911-b3c5-476a14bc809a-kube-api-access-gw9rs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.066588 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.066568    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.089810 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.089790    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:26.130990 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.129221    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"]
Oct 30 19:53:26.134415 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.134398    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.165588 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.165557    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"]
Oct 30 19:53:26.241524 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.241492    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.241676 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.241545    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.241717 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.241667    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.241717 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.241702    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhvmx\" (UniqueName: \"kubernetes.io/projected/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-kube-api-access-xhvmx\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.241848 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.241830    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.241903 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.241891    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.241973 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.241961    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.242007 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.241987    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.242067 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.242014    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.342927 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.342894    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhvmx\" (UniqueName: \"kubernetes.io/projected/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-kube-api-access-xhvmx\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.343125 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.342954    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.343125 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.343002    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.343125 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.343096    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.343125 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.343114    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.343363 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.343139    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.343363 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.343167    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.343363 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.343206    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.343363 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.343277    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.343363 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.343339    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.343554 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.343415    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.343554 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.343526    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.343649 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.343621    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.343903 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.343886    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.345321 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.345299    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.345533 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.345517    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.361579 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.361560    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.362046 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.361997    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhvmx\" (UniqueName: \"kubernetes.io/projected/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-kube-api-access-xhvmx\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.406524 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.406491    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"]
Oct 30 19:53:26.408553 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:53:26.408532    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3a55c8_98fe_4911_b3c5_476a14bc809a.slice/crio-8d4d00a91146b60c238ae868b235e643a252a4d1db0157af7e91e5d117415c11 WatchSource:0}: Error finding container 8d4d00a91146b60c238ae868b235e643a252a4d1db0157af7e91e5d117415c11: Status 404 returned error can't find the container with id 8d4d00a91146b60c238ae868b235e643a252a4d1db0157af7e91e5d117415c11
Oct 30 19:53:26.411350 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.411320    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:26.411423 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.411383    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:26.411423 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.411411    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:26.456613 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.456590    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:26.756497 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.756475    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"]
Oct 30 19:53:26.758306 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:53:26.758281    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7ec4d2_c450_42d0_aebe_8bbf4b8832fe.slice/crio-3af2c3780707ee48c5e82880904a4e71001cf20105664b33d8fd374cf19745fd WatchSource:0}: Error finding container 3af2c3780707ee48c5e82880904a4e71001cf20105664b33d8fd374cf19745fd: Status 404 returned error can't find the container with id 3af2c3780707ee48c5e82880904a4e71001cf20105664b33d8fd374cf19745fd
Oct 30 19:53:26.760856 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.760824    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:26.760942 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.760888    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:26.760942 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.760917    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:26.865088 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.850382    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml" event={"ID":"2d3a55c8-98fe-4911-b3c5-476a14bc809a","Type":"ContainerStarted","Data":"51a2f0fab52732b11f2ce8b7ebae38bedff6c786d990b19419cff21d5f8f9dba"}
Oct 30 19:53:26.865088 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.850447    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml" event={"ID":"2d3a55c8-98fe-4911-b3c5-476a14bc809a","Type":"ContainerStarted","Data":"8d4d00a91146b60c238ae868b235e643a252a4d1db0157af7e91e5d117415c11"}
Oct 30 19:53:26.865088 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.853333    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7" event={"ID":"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe","Type":"ContainerStarted","Data":"3af2c3780707ee48c5e82880904a4e71001cf20105664b33d8fd374cf19745fd"}
Oct 30 19:53:26.928952 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:26.928894    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml" podStartSLOduration=1.928876291 podStartE2EDuration="1.928876291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:26.920634044 +0000 UTC m=+1361.504903661" watchObservedRunningTime="2026-10-30 19:53:26.928876291 +0000 UTC m=+1361.513145907"
Oct 30 19:53:27.090208 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:27.090141    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:27.091281 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:27.091258    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:53:27.091369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:27.091304    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml" podUID="2d3a55c8-98fe-4911-b3c5-476a14bc809a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 19:53:27.298750 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:27.298723    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:27.337185 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:27.337153    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:27.871398 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:27.871360    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7" event={"ID":"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe","Type":"ContainerStarted","Data":"3ce0d393151d17b2bf171fd0af4188458c3eb1b356a174ac9db0248abb57e381"}
Oct 30 19:53:27.885150 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:27.885119    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:53:27.933050 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:27.932957    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7" podStartSLOduration=1.932935756 podStartE2EDuration="1.932935756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:27.926540667 +0000 UTC m=+1362.510810289" watchObservedRunningTime="2026-10-30 19:53:27.932935756 +0000 UTC m=+1362.517205373"
Oct 30 19:53:28.090945 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:28.090908    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:53:28.091317 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:28.090973    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml" podUID="2d3a55c8-98fe-4911-b3c5-476a14bc809a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 19:53:28.457744 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:28.457629    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:28.458747 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:28.458715    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:53:28.458891 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:28.458873    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7" podUID="4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 19:53:29.091156 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:29.090995    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:53:29.091156 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:29.091068    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml" podUID="2d3a55c8-98fe-4911-b3c5-476a14bc809a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 19:53:29.461465 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:29.461165    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:53:29.461465 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:29.461222    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7" podUID="4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 19:53:30.090610 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:30.090579    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:53:30.090784 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:30.090632    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml" podUID="2d3a55c8-98fe-4911-b3c5-476a14bc809a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 19:53:30.457853 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:30.457754    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:53:30.457853 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:30.457825    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7" podUID="4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 19:53:31.090581 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:31.090549    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:53:31.090750 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:31.090611    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml" podUID="2d3a55c8-98fe-4911-b3c5-476a14bc809a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 19:53:31.458473 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:31.458379    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:53:31.459090 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:31.459023    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7" podUID="4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 19:53:32.090731 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:32.090696    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:53:32.090900 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:32.090751    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml" podUID="2d3a55c8-98fe-4911-b3c5-476a14bc809a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 19:53:32.458019 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:32.457941    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:53:32.458274 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:32.458242    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7" podUID="4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 19:53:33.090506 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:33.090460    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:53:33.090905 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:33.090525    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml" podUID="2d3a55c8-98fe-4911-b3c5-476a14bc809a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 19:53:33.458075 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:33.457960    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:53:33.458241 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:33.458073    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7" podUID="4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 19:53:33.599918 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:33.599848    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:33.619398 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:33.619361    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:33.922213 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:33.922182    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:53:34.239111 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:34.223258    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:34.239111 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:34.231460    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:34.250729 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:34.250656    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 19:53:35.088834 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:35.088804    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:35.090159 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:35.090138    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:35.094297 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:35.094242    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 19:53:39.660112 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:53:39.660070    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:53:40.568411 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.568376    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"]
Oct 30 19:53:40.573833 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.573811    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.589883 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.589852    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"]
Oct 30 19:53:40.702634 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.702602    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.703073 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.702641    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.703073 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.702839    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.703073 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.702883    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.703073 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.702916    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.703223 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.703087    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.703223 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.703200    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9chj\" (UniqueName: \"kubernetes.io/projected/f9581298-8aed-41b0-873c-10b0ffe14c27-kube-api-access-n9chj\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.703317 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.703283    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9581298-8aed-41b0-873c-10b0ffe14c27-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.703976 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.703953    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.809052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.806789    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.809052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.806865    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.809052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.806900    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.809052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.807061    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.809052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.807095    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.809052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.807130    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.809052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.807177    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.809052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.807326    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9chj\" (UniqueName: \"kubernetes.io/projected/f9581298-8aed-41b0-873c-10b0ffe14c27-kube-api-access-n9chj\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.809052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.807393    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9581298-8aed-41b0-873c-10b0ffe14c27-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.809052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.808227    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9581298-8aed-41b0-873c-10b0ffe14c27-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.809052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.808848    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.809744 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.809124    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.809744 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.809373    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.810700 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.810673    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.817513 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.817484    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.819380 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.819322    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.824009 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.823986    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.831223 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.831195    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9chj\" (UniqueName: \"kubernetes.io/projected/f9581298-8aed-41b0-873c-10b0ffe14c27-kube-api-access-n9chj\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:40.899495 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:40.899425    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:41.198246 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:41.198222    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"]
Oct 30 19:53:41.200373 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:53:41.200346    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9581298_8aed_41b0_873c_10b0ffe14c27.slice/crio-15c2a91f1dc80da32e911b3097e592a885977e73066c8e2330fb610c834ca2d5 WatchSource:0}: Error finding container 15c2a91f1dc80da32e911b3097e592a885977e73066c8e2330fb610c834ca2d5: Status 404 returned error can't find the container with id 15c2a91f1dc80da32e911b3097e592a885977e73066c8e2330fb610c834ca2d5
Oct 30 19:53:41.202881 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:41.202851    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:41.202957 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:41.202925    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:41.202997 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:41.202954    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:42.016085 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:42.015115    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj" event={"ID":"f9581298-8aed-41b0-873c-10b0ffe14c27","Type":"ContainerStarted","Data":"02744e51d9d6add4e5b2da1eacd16e7d27b2cd056ed6feff6a960e7f0a3c453b"}
Oct 30 19:53:42.016085 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:42.015184    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj" event={"ID":"f9581298-8aed-41b0-873c-10b0ffe14c27","Type":"ContainerStarted","Data":"15c2a91f1dc80da32e911b3097e592a885977e73066c8e2330fb610c834ca2d5"}
Oct 30 19:53:42.048347 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:42.048295    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj" podStartSLOduration=2.048280539 podStartE2EDuration="2.048280539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:42.044924798 +0000 UTC m=+1376.629194428" watchObservedRunningTime="2026-10-30 19:53:42.048280539 +0000 UTC m=+1376.632550154"
Oct 30 19:53:42.900605 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:42.900571    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:42.906065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:42.902552    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:53:42.906065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:42.902603    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj" podUID="f9581298-8aed-41b0-873c-10b0ffe14c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 19:53:43.899895 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:43.899861    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:53:43.900384 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:43.899928    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj" podUID="f9581298-8aed-41b0-873c-10b0ffe14c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 19:53:44.900697 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:44.900660    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:53:44.901103 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:44.900730    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj" podUID="f9581298-8aed-41b0-873c-10b0ffe14c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 19:53:45.900235 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:45.900197    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:53:45.900407 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:45.900273    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj" podUID="f9581298-8aed-41b0-873c-10b0ffe14c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 19:53:46.900276 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:46.900240    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:53:46.900789 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:46.900303    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj" podUID="f9581298-8aed-41b0-873c-10b0ffe14c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 19:53:47.900063 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:47.900011    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:53:47.900264 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:47.900091    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj" podUID="f9581298-8aed-41b0-873c-10b0ffe14c27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 19:53:48.904946 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:48.904920    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:48.905407 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:48.905188    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:48.905990 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:48.905970    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 19:53:55.100722 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.100692    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"]
Oct 30 19:53:55.114006 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.113971    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.138469 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.138443    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"]
Oct 30 19:53:55.169137 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.169104    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.169296 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.169243    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.169390 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.169372    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.169460 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.169439    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkjc9\" (UniqueName: \"kubernetes.io/projected/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-kube-api-access-nkjc9\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.169518 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.169481    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.169622 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.169608    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.169686 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.169670    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.169753 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.169734    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.169807 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.169781    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.270616 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.270586    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.270616 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.270623    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.270851 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.270650    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.270851 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.270703    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.270851 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.270775    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.270851 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.270834    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.271104 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.270867    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkjc9\" (UniqueName: \"kubernetes.io/projected/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-kube-api-access-nkjc9\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.271104 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.270898    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.271104 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.271000    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.271104 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.271067    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.271295 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.271146    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.271422 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.271396    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.271557 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.271403    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.271643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.271621    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.273270 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.273235    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.273577 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.273557    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.296335 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.296311    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.297749 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.297728    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkjc9\" (UniqueName: \"kubernetes.io/projected/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-kube-api-access-nkjc9\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.331477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.331450    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"]
Oct 30 19:53:55.338265 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.338241    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.360742 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.360681    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"]
Oct 30 19:53:55.372004 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.371970    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.372153 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.372137    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.372210 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.372166    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.372210 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.372191    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c73d1b-a864-4534-944a-a543d04a2c3b-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.372383 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.372359    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.372439 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.372421    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kp6t\" (UniqueName: \"kubernetes.io/projected/a8c73d1b-a864-4534-944a-a543d04a2c3b-kube-api-access-7kp6t\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.372509 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.372492    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.372603 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.372586    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.372682 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.372664    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.429893 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.429865    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:55.474411 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.474375    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.474599 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.474432    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.474599 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.474507    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.474599 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.474521    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.474599 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.474536    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c73d1b-a864-4534-944a-a543d04a2c3b-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.474599 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.474582    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.474863 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.474615    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kp6t\" (UniqueName: \"kubernetes.io/projected/a8c73d1b-a864-4534-944a-a543d04a2c3b-kube-api-access-7kp6t\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.474863 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.474659    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.474863 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.474700    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.475129 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.475013    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.475496 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.475473    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.476289 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.475942    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.476289 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.476274    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c73d1b-a864-4534-944a-a543d04a2c3b-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.476569 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.476544    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.478564 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.478539    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.480389 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.480371    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.501646 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.500993    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kp6t\" (UniqueName: \"kubernetes.io/projected/a8c73d1b-a864-4534-944a-a543d04a2c3b-kube-api-access-7kp6t\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.510434 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.510406    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.668099 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.654287    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:55.732159 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.730874    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"]
Oct 30 19:53:55.737873 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.737818    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:55.737969 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.737916    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:55.738075 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.737964    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:55.967588 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.967563    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"]
Oct 30 19:53:55.969881 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:53:55.969850    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c73d1b_a864_4534_944a_a543d04a2c3b.slice/crio-112a9f7f1001bb160f7a61dc8d64341cd99bf49a6db32e9ae694e18b423e77d4 WatchSource:0}: Error finding container 112a9f7f1001bb160f7a61dc8d64341cd99bf49a6db32e9ae694e18b423e77d4: Status 404 returned error can't find the container with id 112a9f7f1001bb160f7a61dc8d64341cd99bf49a6db32e9ae694e18b423e77d4
Oct 30 19:53:55.972709 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.972675    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:55.972797 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.972741    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:55.972797 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:55.972775    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:56.013362 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.013336    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"]
Oct 30 19:53:56.020080 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.020060    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.065870 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.065704    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"]
Oct 30 19:53:56.092252 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.092223    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.092376 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.092318    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.092376 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.092369    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.092495 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.092412    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.092495 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.092440    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.092495 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.092463    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.092495 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.092481    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.092697 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.092508    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7sww\" (UniqueName: \"kubernetes.io/projected/d7619cb9-6eef-4207-a4ee-a892f1920ff6-kube-api-access-d7sww\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.092697 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.092598    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.095147 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.095122    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"]
Oct 30 19:53:56.105185 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.105153    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.110283 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.110239    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"]
Oct 30 19:53:56.138921 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.136675    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84" event={"ID":"a8c73d1b-a864-4534-944a-a543d04a2c3b","Type":"ContainerStarted","Data":"476f128b0efd88bdb29619b29dc2a41c8ed9544f81491afd76a1430d130eee24"}
Oct 30 19:53:56.138921 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.136723    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84" event={"ID":"a8c73d1b-a864-4534-944a-a543d04a2c3b","Type":"ContainerStarted","Data":"112a9f7f1001bb160f7a61dc8d64341cd99bf49a6db32e9ae694e18b423e77d4"}
Oct 30 19:53:56.141500 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.139211    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq" event={"ID":"688f0cb2-8d4a-47ab-ab79-fb1a54109e48","Type":"ContainerStarted","Data":"7d7805b2830bcc21f460e41c7be2ef41ba2d2357171e84e806496ca083d7a23c"}
Oct 30 19:53:56.141500 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.139260    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq" event={"ID":"688f0cb2-8d4a-47ab-ab79-fb1a54109e48","Type":"ContainerStarted","Data":"8066dc67c56eac11e272c0d50fde3008ae2f9a7da9ba79c25580890cc07da618"}
Oct 30 19:53:56.193718 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.193687    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.193718 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.193747    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.193718 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.193773    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.194130 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.193788    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.194130 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.193807    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.194130 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.193866    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.194130 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.193897    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.194130 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.193919    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.194130 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.193934    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.194130 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.193957    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhd6n\" (UniqueName: \"kubernetes.io/projected/3086a82d-b9ca-4c66-8b65-a1eb764909e5-kube-api-access-bhd6n\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.194130 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.193983    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7sww\" (UniqueName: \"kubernetes.io/projected/d7619cb9-6eef-4207-a4ee-a892f1920ff6-kube-api-access-d7sww\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.194130 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.194011    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.194931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.194727    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.194931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.194733    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.194931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.194851    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.194931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.194899    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.195201 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.194984    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.195201 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.195099    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.195201 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.195144    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.195362 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.195243    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.196055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.195995    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.196527 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.196502    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.196611 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.196529    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.198334 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.198306    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.198606 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.198586    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.222986 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.222907    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq" podStartSLOduration=1.222888158 podStartE2EDuration="1.222888158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:56.219561794 +0000 UTC m=+1390.803831434" watchObservedRunningTime="2026-10-30 19:53:56.222888158 +0000 UTC m=+1390.807157774"
Oct 30 19:53:56.225382 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.225351    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.233078 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.230888    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7sww\" (UniqueName: \"kubernetes.io/projected/d7619cb9-6eef-4207-a4ee-a892f1920ff6-kube-api-access-d7sww\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.273469 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.273421    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84" podStartSLOduration=1.273402332 podStartE2EDuration="1.273402332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:56.272554474 +0000 UTC m=+1390.856824091" watchObservedRunningTime="2026-10-30 19:53:56.273402332 +0000 UTC m=+1390.857671954"
Oct 30 19:53:56.296672 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.296635    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.296812 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.296676    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.296812 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.296693    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.296812 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.296786    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhd6n\" (UniqueName: \"kubernetes.io/projected/3086a82d-b9ca-4c66-8b65-a1eb764909e5-kube-api-access-bhd6n\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.296982 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.296832    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.296982 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.296864    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.296982 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.296898    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.297179 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.296987    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.297179 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.297065    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.297179 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.297109    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.297365 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.297208    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.297511 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.297488    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.297740 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.297721    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.297920 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.297902    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.299457 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.299422    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.299982 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.299962    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.320719 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.320694    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhd6n\" (UniqueName: \"kubernetes.io/projected/3086a82d-b9ca-4c66-8b65-a1eb764909e5-kube-api-access-bhd6n\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.322870 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.322849    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.372342 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.372314    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:56.430016 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.429981    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:53:56.431649 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.431619    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:53:56.431774 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.431682    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq" podUID="688f0cb2-8d4a-47ab-ab79-fb1a54109e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 19:53:56.448070 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.438364    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:56.654896 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.654864    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:53:56.670414 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.670388    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"]
Oct 30 19:53:56.672346 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:53:56.672320    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7619cb9_6eef_4207_a4ee_a892f1920ff6.slice/crio-a2ada82e4a18843045c06dc6ef0f7c69e2d30ed4aeeb7272fbfa64740e497406 WatchSource:0}: Error finding container a2ada82e4a18843045c06dc6ef0f7c69e2d30ed4aeeb7272fbfa64740e497406: Status 404 returned error can't find the container with id a2ada82e4a18843045c06dc6ef0f7c69e2d30ed4aeeb7272fbfa64740e497406
Oct 30 19:53:56.674929 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.674895    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:56.674998 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.674959    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:56.674998 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.674988    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:56.754722 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.748706    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"]
Oct 30 19:53:56.756668 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.756597    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:56.756847 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.756703    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:56.756847 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:56.756747    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:53:57.251555 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:57.251517    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" event={"ID":"3086a82d-b9ca-4c66-8b65-a1eb764909e5","Type":"ContainerStarted","Data":"d9cfd4bbda33519067b37f3ca65567fabdac4388908769106429df467c2134e4"}
Oct 30 19:53:57.252149 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:57.252123    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" event={"ID":"3086a82d-b9ca-4c66-8b65-a1eb764909e5","Type":"ContainerStarted","Data":"eb495302a0cc801b6f90761d0e74f0489c3f339019faabef03ffaf64adecddd4"}
Oct 30 19:53:57.293049 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:57.283446    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" event={"ID":"d7619cb9-6eef-4207-a4ee-a892f1920ff6","Type":"ContainerStarted","Data":"7bfb4ea8152e81b8a525bc6c87d2bc87c89ddf059095af814485d668e6c4f981"}
Oct 30 19:53:57.293049 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:57.283496    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" event={"ID":"d7619cb9-6eef-4207-a4ee-a892f1920ff6","Type":"ContainerStarted","Data":"a2ada82e4a18843045c06dc6ef0f7c69e2d30ed4aeeb7272fbfa64740e497406"}
Oct 30 19:53:57.293049 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:57.289484    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" podStartSLOduration=1.289465949 podStartE2EDuration="1.289465949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:57.286976227 +0000 UTC m=+1391.871245844" watchObservedRunningTime="2026-10-30 19:53:57.289465949 +0000 UTC m=+1391.873735570"
Oct 30 19:53:57.377188 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:57.377091    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:53:57.377408 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:57.377192    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:53:57.377408 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:57.377255    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 19:53:57.444222 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:57.440388    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:53:57.444222 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:57.440882    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:53:57.444222 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:57.440942    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq" podUID="688f0cb2-8d4a-47ab-ab79-fb1a54109e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 19:53:57.445011 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:57.444899    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:53:57.445011 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:57.444950    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 19:53:57.662709 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:57.662620    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:53:57.662709 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:57.662671    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84" podUID="a8c73d1b-a864-4534-944a-a543d04a2c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 19:53:58.373562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:58.373511    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:53:58.374175 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:58.374143    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 19:53:58.459402 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:58.455718    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:53:58.459402 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:58.455774    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 19:53:58.459402 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:58.456219    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:53:58.459402 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:58.456259    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq" podUID="688f0cb2-8d4a-47ab-ab79-fb1a54109e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 19:53:58.663361 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:58.661994    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:53:58.671048 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:58.669095    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84" podUID="a8c73d1b-a864-4534-944a-a543d04a2c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 19:53:59.375056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:59.374389    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:53:59.375056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:59.374447    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 19:53:59.436528 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:59.436491    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:53:59.436809 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:59.436760    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq" podUID="688f0cb2-8d4a-47ab-ab79-fb1a54109e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 19:53:59.441381 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:59.441353    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:53:59.441558 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:59.441536    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 19:53:59.503160 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:59.503113    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:59.654996 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:59.654918    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:53:59.655255 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:59.655225    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84" podUID="a8c73d1b-a864-4534-944a-a543d04a2c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 19:53:59.874176 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:59.874118    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:59.903166 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:59.903121    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:59.928114 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:53:59.903738    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:00.036052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.035117    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:54:00.376885 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.376850    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:54:00.402976 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.402929    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:00.434135 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.403218    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 19:54:00.434374 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.377388    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:00.447060 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.443590    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:54:00.447060 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.443655    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq" podUID="688f0cb2-8d4a-47ab-ab79-fb1a54109e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 19:54:00.447060 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.444223    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:54:00.447060 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.444263    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 19:54:00.514773 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.514733    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:00.521540 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.521511    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:54:00.564288 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.564204    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" podStartSLOduration=5.5641829210000004 podStartE2EDuration="5.564182921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:57.33841572 +0000 UTC m=+1391.922685364" watchObservedRunningTime="2026-10-30 19:54:00.564182921 +0000 UTC m=+1395.148452538"
Oct 30 19:54:00.726154 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.719997    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:54:00.726154 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.720083    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84" podUID="a8c73d1b-a864-4534-944a-a543d04a2c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 19:54:00.726154 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.722763    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:00.903103 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.889120    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:00.903103 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.889577    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:54:00.919093 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.903618    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:00.919093 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.904073    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:54:00.969077 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.951439    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:00.969077 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.951859    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:00.969077 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:00.952207    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:54:01.121090 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.120455    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:54:01.121090 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.120666    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:54:01.162218 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.161994    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:54:01.168060 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.164772    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:54:01.289437 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.284775    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:54:01.420939 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.420850    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:54:01.421537 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.421491    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 19:54:01.437976 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.437944    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:54:01.438232 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.438205    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq" podUID="688f0cb2-8d4a-47ab-ab79-fb1a54109e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 19:54:01.448245 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.448216    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:54:01.448565 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.448538    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 19:54:01.563373 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.563236    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:01.621853 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.619572    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:01.649501 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.649475    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:54:01.691963 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.691882    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:54:01.692229 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.692189    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84" podUID="a8c73d1b-a864-4534-944a-a543d04a2c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 19:54:01.988051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.983704    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:01.988051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.985121    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:54:01.988051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:01.986998    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:54:02.001725 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:02.001676    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:02.286218 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:02.286121    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:02.286767 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:02.286420    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:02.389016 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:02.388947    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:54:02.389301 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:02.389251    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 19:54:02.431624 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:02.431588    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:54:02.432276 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:02.432222    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq" podUID="688f0cb2-8d4a-47ab-ab79-fb1a54109e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 19:54:02.440427 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:02.440388    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:54:02.440633 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:02.440609    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 19:54:02.470204 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:02.470152    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:02.659609 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:02.659516    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:54:02.659890 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:02.659846    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84" podUID="a8c73d1b-a864-4534-944a-a543d04a2c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 19:54:02.985251 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:02.985145    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:03.016117 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:03.011491    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:54:03.168974 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:03.168923    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:03.288055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:03.287170    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:54:03.288055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:03.287717    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:54:03.375892 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:03.375854    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:54:03.377683 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:03.377645    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 19:54:03.433256 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:03.433207    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:54:03.475182 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:03.449719    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq" podUID="688f0cb2-8d4a-47ab-ab79-fb1a54109e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 19:54:03.545872 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:03.449644    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:54:03.545872 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:03.536303    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 19:54:03.672062 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:03.671132    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:54:03.672062 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:03.671201    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84" podUID="a8c73d1b-a864-4534-944a-a543d04a2c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 19:54:03.765354 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:03.765311    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:03.889891 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:03.889809    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:54:03.889891 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:03.889872    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:54:03.890135 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:03.889904    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:54:04.376582 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:04.375547    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:54:04.376582 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:04.375609    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 19:54:04.430769 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:04.430734    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:54:04.431023 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:04.430995    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq" podUID="688f0cb2-8d4a-47ab-ab79-fb1a54109e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 19:54:04.440398 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:04.440368    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:54:04.440871 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:04.440842    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 19:54:04.655844 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:04.655761    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:54:04.656129 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:04.656097    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84" podUID="a8c73d1b-a864-4534-944a-a543d04a2c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 19:54:05.373366 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:05.373327    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:54:05.373684 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:05.373637    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 19:54:05.439933 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:05.439901    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:54:05.440189 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:05.440163    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 19:54:05.616044 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:05.616000    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:54:05.617464 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:05.617313    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:54:05.619281 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:05.619249    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 19:54:05.660073 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:05.659974    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:54:05.660500 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:05.660484    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:54:05.661161 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:05.661141    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 19:54:07.379065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:07.377173    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:54:07.379065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:07.377261    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:07.443737 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:07.443693    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:54:07.444296 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:07.444020    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:08.171055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:08.170275    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:08.216209 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:08.216160    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:08.409332 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:08.408220    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:08.426559 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:08.426463    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:54:08.426795 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:08.426769    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:08.455052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:08.450117    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:54:08.455052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:08.450195    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:08.573053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:08.572108    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:08.719233 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:08.719122    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:08.816150 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:08.816108    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:08.839165 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:08.839108    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:08.895317 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:08.895268    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.001081 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:09.000867    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.133061 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:09.132164    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:54:09.133061 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:09.132223    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq" podUID="8fe054e1-8ae6-4066-a04c-94a49bddb6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.459328 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:09.458190    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 19:54:09.459328 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:09.458562    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:54:09.503185 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:09.503147    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:54:09.503447 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:09.503418    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.623144 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:09.579168    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.623144 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:09.604641    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:54:09.623144 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:09.604694    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.629078 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:09.628176    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:54:09.732104 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:09.730641    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.732104 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:09.731514    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:54:09.865060 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:09.822154    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.865060 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:09.822957    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:54:09.865060 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:09.854234    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.865060 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:09.854909    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:54:10.067158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:10.065539    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:54:10.230059 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:10.224255    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:54:10.422063 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:10.421963    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:54:10.422319 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:10.422286    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:10.463098 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:10.459114    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:10.463098 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:10.460596    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:10.610053 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:54:10.591435    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:54:10.610053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:10.593051    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:10.610053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:10.602473    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:54:10.610053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:10.602536    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:10.682249 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:10.682161    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:54:10.719372 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:10.719327    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:10.859129 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:10.855342    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:54:10.920991 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:10.920944    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:11.029301 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.029228    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:54:11.066163 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.066118    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:11.166055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.165109    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:54:11.170053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.166288    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:11.190296 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.190162    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:54:11.190296 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.190227    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" podUID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:11.195125 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.195093    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:54:11.304943 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.303357    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:54:11.304943 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.304875    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:54:11.308873 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.308594    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 19:54:11.456054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.443404    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:54:11.496546 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.480667    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 19:54:11.595923 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.594145    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:11.613445 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.596984    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Oct 30 19:54:11.690401 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.690371    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:54:11.774312 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.774275    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:54:11.775683 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.775457    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:54:11.775683 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.775530    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:11.909676 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.906294    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:54:11.973427 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:11.973392    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:54:12.157425 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:12.157394    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 19:54:25.734382 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:25.734346    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"]
Oct 30 19:54:25.739867 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:25.739849    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:25.761530 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:25.761505    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"]
Oct 30 19:54:25.918454 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:25.918418    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:25.918454 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:25.918456    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88493f4b-7ec6-4f72-92df-03fb99d9c612-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:25.918757 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:25.918558    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:25.918757 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:25.918598    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:25.918757 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:25.918641    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:25.918884 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:25.918781    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6qgw\" (UniqueName: \"kubernetes.io/projected/88493f4b-7ec6-4f72-92df-03fb99d9c612-kube-api-access-z6qgw\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:25.918884 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:25.918811    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:25.918993 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:25.918909    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:25.919080 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:25.919025    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.020650 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.020619    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.020813 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.020678    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.020813 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.020737    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.020813 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.020751    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88493f4b-7ec6-4f72-92df-03fb99d9c612-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.020813 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.020793    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.020977 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.020816    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.020977 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.020858    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.020977 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.020932    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6qgw\" (UniqueName: \"kubernetes.io/projected/88493f4b-7ec6-4f72-92df-03fb99d9c612-kube-api-access-z6qgw\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.021145 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.020977    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.021222 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.021190    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.021486 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.021434    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.021603 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.021549    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.021603 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.021590    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.021685 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.021609    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88493f4b-7ec6-4f72-92df-03fb99d9c612-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.023109 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.023088    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.023197 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.023180    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.039756 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.039708    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6qgw\" (UniqueName: \"kubernetes.io/projected/88493f4b-7ec6-4f72-92df-03fb99d9c612-kube-api-access-z6qgw\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.039997 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.039981    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.054331 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.054307    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:26.361096 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.361068    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"]
Oct 30 19:54:26.362121 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:54:26.362089    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88493f4b_7ec6_4f72_92df_03fb99d9c612.slice/crio-d8bf9b472b8bc63057d3aa6a7dded0450b25490abb361cd3b25a48a7347570ec WatchSource:0}: Error finding container d8bf9b472b8bc63057d3aa6a7dded0450b25490abb361cd3b25a48a7347570ec: Status 404 returned error can't find the container with id d8bf9b472b8bc63057d3aa6a7dded0450b25490abb361cd3b25a48a7347570ec
Oct 30 19:54:26.364955 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.364909    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:26.365068 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.365006    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:26.365135 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.365075    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:26.530369 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.530332    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"]
Oct 30 19:54:26.535076 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.535004    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.564082 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.563932    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"]
Oct 30 19:54:26.637832 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.637748    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"]
Oct 30 19:54:26.644108 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.642092    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.674065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.672452    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"]
Oct 30 19:54:26.731715 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.731682    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.731715 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.731715    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j52d2\" (UniqueName: \"kubernetes.io/projected/8b8025ef-9bd4-463a-bc56-50d8d46673c2-kube-api-access-j52d2\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.731946 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.731924    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.732010 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.731985    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.732088 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.732013    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.732130 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.732090    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.732188 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.732142    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.732238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.732205    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.732312 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.732297    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.833753 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.833719    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.834239 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.833782    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.834239 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.833806    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.834239 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.833840    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.834239 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.833991    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.834239 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.834046    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgbc9\" (UniqueName: \"kubernetes.io/projected/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-kube-api-access-mgbc9\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.834239 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.834138    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.834239 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.834180    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.834646 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.834276    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.834646 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.834285    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.834646 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.834316    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.834646 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.834360    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.834646 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.834398    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.834646 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.834446    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.834646 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.834557    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.834646 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.834585    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.834646 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.834601    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.836502 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.834650    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.836502 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.834697    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.836502 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.834972    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.836502 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.835416    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j52d2\" (UniqueName: \"kubernetes.io/projected/8b8025ef-9bd4-463a-bc56-50d8d46673c2-kube-api-access-j52d2\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.836502 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.835502    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.836502 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.835734    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.836727 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.836564    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.837246 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.837224    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.876841 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.869192    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.876841 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.869491    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j52d2\" (UniqueName: \"kubernetes.io/projected/8b8025ef-9bd4-463a-bc56-50d8d46673c2-kube-api-access-j52d2\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:26.936989 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.936907    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.936989 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.936942    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgbc9\" (UniqueName: \"kubernetes.io/projected/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-kube-api-access-mgbc9\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.936989 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.936973    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.937356 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.936996    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.937356 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.937062    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.937356 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.937160    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.937356 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.937220    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.937356 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.937271    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.937356 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.937342    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.937648 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.937505    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.937648 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.937563    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.937734 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.937684    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.937734 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.937713    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.937959 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.937937    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.939626 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.939607    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.939626 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.939618    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.965442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.965407    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.965632 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.965614    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgbc9\" (UniqueName: \"kubernetes.io/projected/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-kube-api-access-mgbc9\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:26.969477 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:26.969457    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:27.151221 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:27.151106    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:27.253640 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:27.253597    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg" event={"ID":"88493f4b-7ec6-4f72-92df-03fb99d9c612","Type":"ContainerStarted","Data":"740036cd4ff7c7c2f138fd5a15eea5bf22eb69a7c272e192f8fec8ff9e72da9f"}
Oct 30 19:54:27.253836 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:27.253674    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg" event={"ID":"88493f4b-7ec6-4f72-92df-03fb99d9c612","Type":"ContainerStarted","Data":"d8bf9b472b8bc63057d3aa6a7dded0450b25490abb361cd3b25a48a7347570ec"}
Oct 30 19:54:27.301801 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:27.301741    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg" podStartSLOduration=2.301725822 podStartE2EDuration="2.301725822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:27.297554096 +0000 UTC m=+1421.881823725" watchObservedRunningTime="2026-10-30 19:54:27.301725822 +0000 UTC m=+1421.885995436"
Oct 30 19:54:27.344062 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:27.341974    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"]
Oct 30 19:54:27.348076 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:54:27.346999    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5fa6c2_e2dc_43d5_bdae_a0bb0c9312ec.slice/crio-083ee319d2ebf93d5559c20db402a670f9b70cd27222ab654c72d5a364aa156a WatchSource:0}: Error finding container 083ee319d2ebf93d5559c20db402a670f9b70cd27222ab654c72d5a364aa156a: Status 404 returned error can't find the container with id 083ee319d2ebf93d5559c20db402a670f9b70cd27222ab654c72d5a364aa156a
Oct 30 19:54:27.350166 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:27.350134    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:27.350273 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:27.350209    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:27.350273 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:27.350240    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:27.542080 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:54:27.542050    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8025ef_9bd4_463a_bc56_50d8d46673c2.slice/crio-70cd9538a3631d7e6bf628b2b4bfd954e9fa0234e7dac7aa7a53f49dc2a50e33 WatchSource:0}: Error finding container 70cd9538a3631d7e6bf628b2b4bfd954e9fa0234e7dac7aa7a53f49dc2a50e33: Status 404 returned error can't find the container with id 70cd9538a3631d7e6bf628b2b4bfd954e9fa0234e7dac7aa7a53f49dc2a50e33
Oct 30 19:54:27.544811 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:27.544776    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:27.544918 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:27.544862    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:27.544918 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:27.544908    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:27.546366 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:27.546345    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"]
Oct 30 19:54:28.055348 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:28.055311    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:28.056608 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:28.056583    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:54:28.056727 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:28.056636    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg" podUID="88493f4b-7ec6-4f72-92df-03fb99d9c612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:54:28.260582 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:28.260548    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf" event={"ID":"8b8025ef-9bd4-463a-bc56-50d8d46673c2","Type":"ContainerStarted","Data":"f2ff12c85c16935d05433a328443cb28a184fb0085b4a5f1bb079f59bc6dc24c"}
Oct 30 19:54:28.260799 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:28.260612    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf" event={"ID":"8b8025ef-9bd4-463a-bc56-50d8d46673c2","Type":"ContainerStarted","Data":"70cd9538a3631d7e6bf628b2b4bfd954e9fa0234e7dac7aa7a53f49dc2a50e33"}
Oct 30 19:54:28.262227 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:28.262200    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn" event={"ID":"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec","Type":"ContainerStarted","Data":"e80db79493b688707ecb8b39bb6f64951de6049ad16542d3f4de9c90b0bde2d2"}
Oct 30 19:54:28.262353 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:28.262247    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn" event={"ID":"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec","Type":"ContainerStarted","Data":"083ee319d2ebf93d5559c20db402a670f9b70cd27222ab654c72d5a364aa156a"}
Oct 30 19:54:28.334177 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:28.321874    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf" podStartSLOduration=2.321855426 podStartE2EDuration="2.321855426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:28.320430363 +0000 UTC m=+1422.904699982" watchObservedRunningTime="2026-10-30 19:54:28.321855426 +0000 UTC m=+1422.906125045"
Oct 30 19:54:28.369415 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:28.369357    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn" podStartSLOduration=2.369343426 podStartE2EDuration="2.369343426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:28.365913905 +0000 UTC m=+1422.950183525" watchObservedRunningTime="2026-10-30 19:54:28.369343426 +0000 UTC m=+1422.953613046"
Oct 30 19:54:28.979059 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:28.970388    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:28.983057 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:28.981171    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:54:28.983057 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:28.981257    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn" podUID="9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:54:29.061543 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:29.061505    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:54:29.062129 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:29.062097    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg" podUID="88493f4b-7ec6-4f72-92df-03fb99d9c612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:54:29.173054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:29.162061    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:29.185060 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:29.178624    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:54:29.185060 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:29.178693    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf" podUID="8b8025ef-9bd4-463a-bc56-50d8d46673c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 19:54:29.992064 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:29.988470    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:54:29.992064 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:29.988577    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn" podUID="9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:54:30.070053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:30.069202    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:54:30.070053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:30.069303    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg" podUID="88493f4b-7ec6-4f72-92df-03fb99d9c612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:54:30.156056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:30.155438    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:54:30.156056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:30.155512    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf" podUID="8b8025ef-9bd4-463a-bc56-50d8d46673c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 19:54:30.977055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:30.976395    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:54:30.977055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:30.976459    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn" podUID="9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:54:31.082045 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:31.081994    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:54:31.088130 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:31.088094    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg" podUID="88493f4b-7ec6-4f72-92df-03fb99d9c612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:54:31.178059 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:31.167107    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:54:31.178059 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:31.167198    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf" podUID="8b8025ef-9bd4-463a-bc56-50d8d46673c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 19:54:31.979054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:31.978432    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:54:31.979054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:31.978527    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn" podUID="9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:54:32.063709 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:32.063677    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:54:32.064006 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:32.063977    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg" podUID="88493f4b-7ec6-4f72-92df-03fb99d9c612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:54:32.161056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:32.152428    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:54:32.161056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:32.152502    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf" podUID="8b8025ef-9bd4-463a-bc56-50d8d46673c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 19:54:32.970662 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:32.970624    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:54:32.970852 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:32.970685    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn" podUID="9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:54:33.059568 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:33.059531    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:54:33.059839 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:33.059798    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg" podUID="88493f4b-7ec6-4f72-92df-03fb99d9c612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:54:33.154460 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:33.154421    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:54:33.154744 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:33.154717    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf" podUID="8b8025ef-9bd4-463a-bc56-50d8d46673c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 19:54:33.974523 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:33.974473    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:54:33.975130 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:33.975096    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn" podUID="9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:54:34.151783 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:34.151745    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:54:34.151968 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:34.151806    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf" podUID="8b8025ef-9bd4-463a-bc56-50d8d46673c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 19:54:34.254072 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:34.254040    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:34.255479 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:34.255458    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:34.256643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:34.256617    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 19:54:34.970330 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:34.970299    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:54:34.970549 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:34.970347    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn" podUID="9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:54:35.056563 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:35.056515    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11077-56f396"]
Oct 30 19:54:35.061594 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:35.061564    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11077-56f396"
Oct 30 19:54:35.069944 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:35.069919    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xqd5g\""
Oct 30 19:54:35.072118 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:35.072084    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11077-56f396"]
Oct 30 19:54:35.110458 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:35.110418    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vnmr\" (UniqueName: \"kubernetes.io/projected/5c8575da-00c9-4d35-966c-182aa93d2752-kube-api-access-8vnmr\") pod \"test-curl-11077-56f396\" (UID: \"5c8575da-00c9-4d35-966c-182aa93d2752\") " pod="opendatahub/test-curl-11077-56f396"
Oct 30 19:54:35.217889 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:35.217835    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vnmr\" (UniqueName: \"kubernetes.io/projected/5c8575da-00c9-4d35-966c-182aa93d2752-kube-api-access-8vnmr\") pod \"test-curl-11077-56f396\" (UID: \"5c8575da-00c9-4d35-966c-182aa93d2752\") " pod="opendatahub/test-curl-11077-56f396"
Oct 30 19:54:35.234850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:35.234777    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vnmr\" (UniqueName: \"kubernetes.io/projected/5c8575da-00c9-4d35-966c-182aa93d2752-kube-api-access-8vnmr\") pod \"test-curl-11077-56f396\" (UID: \"5c8575da-00c9-4d35-966c-182aa93d2752\") " pod="opendatahub/test-curl-11077-56f396"
Oct 30 19:54:35.383474 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:35.383435    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11077-56f396"
Oct 30 19:54:35.650339 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:35.650308    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:35.651079 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:35.651007    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:35.652215 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:35.652198    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 19:54:35.707548 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:35.707527    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11077-56f396"]
Oct 30 19:54:35.742450 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:35.740207    5408 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:54:35.974071 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:35.973993    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:35.974343 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:35.974308    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:35.975016 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:35.974998    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 19:54:36.359845 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:36.359812    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11077-56f396" event={"ID":"5c8575da-00c9-4d35-966c-182aa93d2752","Type":"ContainerStarted","Data":"7570edbdef282eeddcd738a0e1f43d163210dc3553e28b825909ed187930fb13"}
Oct 30 19:54:38.896063 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:38.888767    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:39.072054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:39.068713    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:39.171701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:39.168133    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:39.171701 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:39.169441    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:39.200055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:39.198115    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:39.608054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:39.603404    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:39.723054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:39.722800    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:39.789110 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:39.788574    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:39.899099 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:39.896105    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:54:39.899099 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:39.896542    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:54:39.988064 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:39.987606    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:40.196114 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:40.195142    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:40.217053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:40.197112    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:40.217053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:40.211106    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:54:40.217053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:40.211438    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:54:40.217389 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:40.217176    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:54:40.217389 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:40.217249    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:54:40.217389 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:40.217304    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:54:40.217389 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:40.217336    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:54:40.386062 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:40.382189    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:40.411054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:40.400134    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:54:40.411054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:40.400213    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:40.701052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:40.604101    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:54:40.701052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:40.604842    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:54:40.746055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:40.745169    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:40.750050 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:40.746444    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:54:40.788058 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:54:40.783858    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:54:40.972242 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:40.792211    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:54:40.979236 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:40.947660    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:40.987067 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:40.980259    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:54:41.008089 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.005101    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:54:41.008089 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.005470    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:54:41.166186 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.166148    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"]
Oct 30 19:54:41.209053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.182137    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.235671 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.209366    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:41.261460 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.209402    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:41.261460 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.229553    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:54:41.261460 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.229600    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:54:41.303451 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.303411    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.303809 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.303783    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrrbx\" (UniqueName: \"kubernetes.io/projected/e8795c70-cfa2-46d3-85a9-3af34e81144d-kube-api-access-mrrbx\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.304136 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.304023    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.307864 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.307842    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.308012 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.307996    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.320655 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.320629    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8795c70-cfa2-46d3-85a9-3af34e81144d-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.321782 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.321757    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.321961 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.321943    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.343185 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.343157    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.463856 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.463822    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.464166 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.464140    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.464344 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.464325    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrrbx\" (UniqueName: \"kubernetes.io/projected/e8795c70-cfa2-46d3-85a9-3af34e81144d-kube-api-access-mrrbx\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.465905 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.465881    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.514357 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.503934    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.514357 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.511795    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.514357 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.511857    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.514357 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.511911    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.514357 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.512529    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8795c70-cfa2-46d3-85a9-3af34e81144d-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.514357 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.513559    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8795c70-cfa2-46d3-85a9-3af34e81144d-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.583051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.579435    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:41.583051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.580696    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.583051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.580751    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.583051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.580831    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.583051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.581822    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.583051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.582368    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.629606 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.629558    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:41.822761 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.727489    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.822761 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.728160    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrrbx\" (UniqueName: \"kubernetes.io/projected/e8795c70-cfa2-46d3-85a9-3af34e81144d-kube-api-access-mrrbx\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.822761 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.757745    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:41.822761 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.760349    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:41.997955 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.987616    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:54:41.997955 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:41.988972    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:54:42.028088 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.020408    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:54:42.462634 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.072250    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:42.533272 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.244332    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:54:42.533272 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.244382    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:54:42.533272 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.244419    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:42.533272 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.299947    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:42.533272 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.300107    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:42.533272 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.423194    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:54:42.533272 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.498920    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:42.533272 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.428589    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"]
Oct 30 19:54:42.782700 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.754706    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:42.782700 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.760225    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:42.827014 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.823436    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"]
Oct 30 19:54:42.871352 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.871316    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"]
Oct 30 19:54:42.871649 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.871632    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:42.977533 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.976776    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0737c2-db49-4cb0-b815-0b2c9560119e-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:42.977533 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.976867    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:42.977533 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.976969    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:42.977533 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.977007    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:42.977533 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.977074    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:42.977533 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.977117    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:42.977533 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.977152    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp86j\" (UniqueName: \"kubernetes.io/projected/0a0737c2-db49-4cb0-b815-0b2c9560119e-kube-api-access-gp86j\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:42.977533 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.977191    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:42.977533 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.977265    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.020093 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.989160    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:43.020093 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:42.990324    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:54:43.085102 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.084069    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.085102 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.084216    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.085102 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.084260    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.085102 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.084333    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.085102 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.084384    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.085102 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.084423    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp86j\" (UniqueName: \"kubernetes.io/projected/0a0737c2-db49-4cb0-b815-0b2c9560119e-kube-api-access-gp86j\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.085102 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.084469    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.085102 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.084552    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.085102 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.084616    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0737c2-db49-4cb0-b815-0b2c9560119e-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.085711 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.085526    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0737c2-db49-4cb0-b815-0b2c9560119e-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.093302 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.086398    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.093302 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.092888    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.093302 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.093171    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.093882 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.093858    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.097005 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.096954    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.118056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.117978    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.132749 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.132490    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.133574 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.133523    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp86j\" (UniqueName: \"kubernetes.io/projected/0a0737c2-db49-4cb0-b815-0b2c9560119e-kube-api-access-gp86j\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.459105 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.422888    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:43.501054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.464533    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:43.501054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.466728    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:54:43.631295 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.624158    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:43.631295 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.625226    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:54:43.674977 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.673680    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:43.756562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.756519    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:43.813071 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.787801    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:43.886068 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:43.877646    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:54:44.022987 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:44.022867    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:44.369711 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:44.369684    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"]
Oct 30 19:54:44.420050 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:44.407550    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:44.420050 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:44.407640    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:44.420050 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:44.407704    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:44.467101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:44.465862    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:44.646270 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:44.646148    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:44.727694 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:44.672481    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:44.727694 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:44.675084    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:54:44.727694 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:44.688890    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:44.727694 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:44.690138    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:44.806461 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:44.805162    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:44.827243 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:44.819795    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:54:45.077666 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:45.039209    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:45.352474 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:45.351546    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:54:45.367845 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:45.367816    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:54:45.400951 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:45.400911    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn" event={"ID":"e8795c70-cfa2-46d3-85a9-3af34e81144d","Type":"ContainerStarted","Data":"7bdc339e1f71928f79fc46b047e426e2f24301be8175fde8010b1089b06e40e7"}
Oct 30 19:54:45.425248 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:45.425203    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:54:45.475687 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:45.475660    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:54:45.663276 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:45.663231    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:45.843708 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:45.843645    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:46.029049 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:46.025872    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:54:46.029319 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:46.026163    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:46.029492 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:46.029479    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:54:46.033590 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:46.029866    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:46.033858 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:46.033825    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:46.070584 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:46.061876    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:46.094189 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:46.077107    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"]
Oct 30 19:54:46.116152 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:46.115966    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:54:46.127066 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:46.127011    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:54:46.150637 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:46.150615    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:54:46.412108 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:46.411508    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp" event={"ID":"0a0737c2-db49-4cb0-b815-0b2c9560119e","Type":"ContainerStarted","Data":"b8c9cb07dc298fa35bc33d55eb0525e6a73d9a90a63989e437547c83dba168fb"}
Oct 30 19:54:46.412108 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:46.411573    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp" event={"ID":"0a0737c2-db49-4cb0-b815-0b2c9560119e","Type":"ContainerStarted","Data":"8c25a41cc53b8adafe3963c4762c908756910503158469ecbbea0e3937e7fcc1"}
Oct 30 19:54:46.414994 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:46.414939    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn" event={"ID":"e8795c70-cfa2-46d3-85a9-3af34e81144d","Type":"ContainerStarted","Data":"2321955da4c770774302758225887963c857aa8900892b8a2cc4d62e86ff00a5"}
Oct 30 19:54:46.438644 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:46.438590    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp" podStartSLOduration=5.438571911 podStartE2EDuration="5.438571911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:46.43634191 +0000 UTC m=+1441.020611521" watchObservedRunningTime="2026-10-30 19:54:46.438571911 +0000 UTC m=+1441.022841526"
Oct 30 19:54:46.443053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:46.443007    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:46.470320 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:46.470271    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn" podStartSLOduration=5.470253861 podStartE2EDuration="5.470253861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:46.464168423 +0000 UTC m=+1441.048438041" watchObservedRunningTime="2026-10-30 19:54:46.470253861 +0000 UTC m=+1441.054523480"
Oct 30 19:54:47.117772 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:47.117677    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:47.118814 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:47.118468    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:54:47.118814 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:47.118537    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn" podUID="e8795c70-cfa2-46d3-85a9-3af34e81144d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 19:54:47.469299 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:47.469190    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 19:54:47.469527 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:47.469498    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp" podUID="0a0737c2-db49-4cb0-b815-0b2c9560119e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 19:54:48.085265 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:48.085221    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:54:48.085462 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:48.085307    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn" podUID="e8795c70-cfa2-46d3-85a9-3af34e81144d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 19:54:48.442433 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:48.442353    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 19:54:48.443467 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:48.442954    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp" podUID="0a0737c2-db49-4cb0-b815-0b2c9560119e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 19:54:49.169050 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:49.164263    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:54:49.169050 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:49.164321    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn" podUID="e8795c70-cfa2-46d3-85a9-3af34e81144d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 19:54:49.452053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:49.444402    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 19:54:49.452053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:49.444463    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp" podUID="0a0737c2-db49-4cb0-b815-0b2c9560119e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 19:54:49.517350 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:49.517295    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11077-56f396" event={"ID":"5c8575da-00c9-4d35-966c-182aa93d2752","Type":"ContainerStarted","Data":"b4e46efb1d331ee0ec1d43ac9a1057f0bfddd202c106f8164d22fb6b502df8f2"}
Oct 30 19:54:49.573018 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:49.572908    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11077-56f396" podStartSLOduration=1.3422321099999999 podStartE2EDuration="14.572889114s" totalImagesPullingTime="13.230657004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:49.570963452 +0000 UTC m=+1444.155233069" watchObservedRunningTime="2026-10-30 19:54:49.572889114 +0000 UTC m=+1444.157158733"
Oct 30 19:54:50.135085 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:50.112618    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:54:50.137821 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:50.112670    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn" podUID="e8795c70-cfa2-46d3-85a9-3af34e81144d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 19:54:50.440850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:50.440756    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 19:54:50.441148 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:50.441092    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp" podUID="0a0737c2-db49-4cb0-b815-0b2c9560119e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 19:54:51.085726 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:51.085300    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:54:51.085726 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:51.085357    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn" podUID="e8795c70-cfa2-46d3-85a9-3af34e81144d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 19:54:51.458960 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:51.458295    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 19:54:51.458960 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:51.458371    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp" podUID="0a0737c2-db49-4cb0-b815-0b2c9560119e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 19:54:51.490598 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:51.490511    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11077-56f396"]
Oct 30 19:54:51.491609 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:51.491579    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11077-56f396"]
Oct 30 19:54:51.676057 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:51.674277    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11077-56f396" podUID="5c8575da-00c9-4d35-966c-182aa93d2752" containerName="test-curl-11077-56f396" containerID="cri-o://b4e46efb1d331ee0ec1d43ac9a1057f0bfddd202c106f8164d22fb6b502df8f2" gracePeriod=2
Oct 30 19:54:51.682544 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:54:51.682501    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11077-56f396\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="5c8575da-00c9-4d35-966c-182aa93d2752" pod="opendatahub/test-curl-11077-56f396"
Oct 30 19:54:51.702087 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:51.700313    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11077-1db447"]
Oct 30 19:54:51.702087 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:51.701126    5408 cpu_manager.go:414] "containerMap: removing container" podUID="5c8575da-00c9-4d35-966c-182aa93d2752" containerName="test-curl-11077-56f396"
Oct 30 19:54:51.702087 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:51.701144    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c8575da-00c9-4d35-966c-182aa93d2752" containerName="test-curl-11077-56f396"
Oct 30 19:54:51.702087 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:51.701343    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c8575da-00c9-4d35-966c-182aa93d2752" containerName="test-curl-11077-56f396"
Oct 30 19:54:51.739110 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:51.729945    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11077-1db447"]
Oct 30 19:54:51.748894 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:51.735274    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11077-1db447"
Oct 30 19:54:51.878056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:51.874679    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lklsr\" (UniqueName: \"kubernetes.io/projected/7a4b4eae-a145-4e7e-b255-14de8325d7a1-kube-api-access-lklsr\") pod \"test-curl-11077-1db447\" (UID: \"7a4b4eae-a145-4e7e-b255-14de8325d7a1\") " pod="opendatahub/test-curl-11077-1db447"
Oct 30 19:54:51.979838 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:51.979806    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lklsr\" (UniqueName: \"kubernetes.io/projected/7a4b4eae-a145-4e7e-b255-14de8325d7a1-kube-api-access-lklsr\") pod \"test-curl-11077-1db447\" (UID: \"7a4b4eae-a145-4e7e-b255-14de8325d7a1\") " pod="opendatahub/test-curl-11077-1db447"
Oct 30 19:54:51.999941 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:51.999840    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lklsr\" (UniqueName: \"kubernetes.io/projected/7a4b4eae-a145-4e7e-b255-14de8325d7a1-kube-api-access-lklsr\") pod \"test-curl-11077-1db447\" (UID: \"7a4b4eae-a145-4e7e-b255-14de8325d7a1\") " pod="opendatahub/test-curl-11077-1db447"
Oct 30 19:54:52.073507 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:52.073470    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11077-1db447"
Oct 30 19:54:52.144629 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:52.144597    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:54:52.153198 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:52.153164    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn" podUID="e8795c70-cfa2-46d3-85a9-3af34e81144d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 19:54:52.443265 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:52.442952    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 19:54:52.443520 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:52.443488    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp" podUID="0a0737c2-db49-4cb0-b815-0b2c9560119e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 19:54:52.764510 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:52.764487    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11077-1db447"]
Oct 30 19:54:52.766993 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:54:52.766964    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a4b4eae_a145_4e7e_b255_14de8325d7a1.slice/crio-45bd6a41f40fd698ef1ba0da3a51d492f68f3144e2e3b9eaf4cf9f2fbe186378 WatchSource:0}: Error finding container 45bd6a41f40fd698ef1ba0da3a51d492f68f3144e2e3b9eaf4cf9f2fbe186378: Status 404 returned error can't find the container with id 45bd6a41f40fd698ef1ba0da3a51d492f68f3144e2e3b9eaf4cf9f2fbe186378
Oct 30 19:54:53.085387 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:53.085280    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:54:53.085387 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:53.085365    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn" podUID="e8795c70-cfa2-46d3-85a9-3af34e81144d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 19:54:53.447237 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:53.447150    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 19:54:53.447861 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:53.447829    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp" podUID="0a0737c2-db49-4cb0-b815-0b2c9560119e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 19:54:53.704868 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:53.704828    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11077-1db447" event={"ID":"7a4b4eae-a145-4e7e-b255-14de8325d7a1","Type":"ContainerStarted","Data":"a298f679c64134197431196131d4bdd4ba528c4dccf0f1366261d80c689db60c"}
Oct 30 19:54:53.705012 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:53.704884    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11077-1db447" event={"ID":"7a4b4eae-a145-4e7e-b255-14de8325d7a1","Type":"ContainerStarted","Data":"45bd6a41f40fd698ef1ba0da3a51d492f68f3144e2e3b9eaf4cf9f2fbe186378"}
Oct 30 19:54:53.733079 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:53.731079    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11077-1db447" podStartSLOduration=2.731060573 podStartE2EDuration="2.731060573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:53.72989991 +0000 UTC m=+1448.314169528" watchObservedRunningTime="2026-10-30 19:54:53.731060573 +0000 UTC m=+1448.315330189"
Oct 30 19:54:54.002946 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:54.002415    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11077-56f396"
Oct 30 19:54:54.074249 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:54.074198    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c8575da-00c9-4d35-966c-182aa93d2752-kube-api-access-8vnmr\" (UniqueName: \"kubernetes.io/projected/5c8575da-00c9-4d35-966c-182aa93d2752-kube-api-access-8vnmr\") pod \"5c8575da-00c9-4d35-966c-182aa93d2752\" (UID: \"5c8575da-00c9-4d35-966c-182aa93d2752\") "
Oct 30 19:54:54.095609 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:54.095181    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:54:54.095609 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:54.095238    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn" podUID="e8795c70-cfa2-46d3-85a9-3af34e81144d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 19:54:54.095609 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:54.095469    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8575da-00c9-4d35-966c-182aa93d2752-kube-api-access-8vnmr" pod "5c8575da-00c9-4d35-966c-182aa93d2752" (UID: "5c8575da-00c9-4d35-966c-182aa93d2752"). InnerVolumeSpecName "kube-api-access-8vnmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:54.195706 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:54.195671    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vnmr\" (UniqueName: \"kubernetes.io/projected/5c8575da-00c9-4d35-966c-182aa93d2752-kube-api-access-8vnmr\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:54.439818 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:54.439733    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 19:54:54.439818 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:54.439803    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp" podUID="0a0737c2-db49-4cb0-b815-0b2c9560119e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 19:54:54.721913 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:54.721827    5408 generic.go:356] "Generic (PLEG): container finished" podID="5c8575da-00c9-4d35-966c-182aa93d2752" containerID="b4e46efb1d331ee0ec1d43ac9a1057f0bfddd202c106f8164d22fb6b502df8f2" exitCode=137
Oct 30 19:54:54.722480 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:54.721896    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11077-56f396"
Oct 30 19:54:54.722989 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:54.721935    5408 scope.go:122] "RemoveContainer" containerID="b4e46efb1d331ee0ec1d43ac9a1057f0bfddd202c106f8164d22fb6b502df8f2"
Oct 30 19:54:54.746570 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:54.746502    5408 scope.go:122] "RemoveContainer" containerID="b4e46efb1d331ee0ec1d43ac9a1057f0bfddd202c106f8164d22fb6b502df8f2"
Oct 30 19:54:54.759170 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:54:54.746863    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e46efb1d331ee0ec1d43ac9a1057f0bfddd202c106f8164d22fb6b502df8f2\": container with ID starting with b4e46efb1d331ee0ec1d43ac9a1057f0bfddd202c106f8164d22fb6b502df8f2 not found: ID does not exist" containerID="b4e46efb1d331ee0ec1d43ac9a1057f0bfddd202c106f8164d22fb6b502df8f2"
Oct 30 19:54:54.759170 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:54.746900    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e46efb1d331ee0ec1d43ac9a1057f0bfddd202c106f8164d22fb6b502df8f2"} err="failed to get container status \"b4e46efb1d331ee0ec1d43ac9a1057f0bfddd202c106f8164d22fb6b502df8f2\": rpc error: code = NotFound desc = could not find container \"b4e46efb1d331ee0ec1d43ac9a1057f0bfddd202c106f8164d22fb6b502df8f2\": container with ID starting with b4e46efb1d331ee0ec1d43ac9a1057f0bfddd202c106f8164d22fb6b502df8f2 not found: ID does not exist"
Oct 30 19:54:54.771745 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:54:54.771707    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11077-56f396\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="5c8575da-00c9-4d35-966c-182aa93d2752" pod="opendatahub/test-curl-11077-56f396"
Oct 30 19:54:54.818971 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:54.815816    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11077-1db447"]
Oct 30 19:54:54.837046 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:54.822628    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11077-1db447"]
Oct 30 19:54:55.268532 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:55.268500    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:55.269130 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:55.269099    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:55.270066 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:55.269981    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 19:54:55.271732 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:54:55.271704    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11077-56f396\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="5c8575da-00c9-4d35-966c-182aa93d2752" pod="opendatahub/test-curl-11077-56f396"
Oct 30 19:54:55.306063 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:54:55.306010    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11077-56f396\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="5c8575da-00c9-4d35-966c-182aa93d2752" pod="opendatahub/test-curl-11077-56f396"
Oct 30 19:54:55.490507 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:55.484366    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:55.523159 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:55.501623    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:55.523159 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:54:55.510842    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11077-56f396\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="5c8575da-00c9-4d35-966c-182aa93d2752" pod="opendatahub/test-curl-11077-56f396"
Oct 30 19:54:55.523159 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:55.521693    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 19:54:55.557955 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:54:55.557916    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11077-56f396\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="5c8575da-00c9-4d35-966c-182aa93d2752" pod="opendatahub/test-curl-11077-56f396"
Oct 30 19:54:55.571854 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:55.571409    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c8575da-00c9-4d35-966c-182aa93d2752" path="/var/lib/kubelet/pods/5c8575da-00c9-4d35-966c-182aa93d2752/volumes"
Oct 30 19:54:55.741988 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:55.741943    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11077-1db447" podUID="7a4b4eae-a145-4e7e-b255-14de8325d7a1" containerName="test-curl-11077-1db447" containerID="cri-o://a298f679c64134197431196131d4bdd4ba528c4dccf0f1366261d80c689db60c" gracePeriod=2
Oct 30 19:54:55.749347 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:54:55.749317    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11077-1db447\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="7a4b4eae-a145-4e7e-b255-14de8325d7a1" pod="opendatahub/test-curl-11077-1db447"
Oct 30 19:54:57.516063 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:57.515402    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj"]
Oct 30 19:54:57.516611 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:57.516470    5408 cpu_manager.go:414] "containerMap: removing container" podUID="7a4b4eae-a145-4e7e-b255-14de8325d7a1" containerName="test-curl-11077-1db447"
Oct 30 19:54:57.516611 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:57.516532    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a4b4eae-a145-4e7e-b255-14de8325d7a1" containerName="test-curl-11077-1db447"
Oct 30 19:54:57.540333 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:57.516790    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a4b4eae-a145-4e7e-b255-14de8325d7a1" containerName="test-curl-11077-1db447"
Oct 30 19:54:57.616602 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:57.616566    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj"]
Oct 30 19:54:57.651490 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:57.618156    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj"
Oct 30 19:54:57.703057 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:57.693692    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ea210528\""
Oct 30 19:54:57.728723 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:57.728685    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7sp8\" (UniqueName: \"kubernetes.io/projected/3de3a976-5be0-4ba7-a869-4c58bb8fee0f-kube-api-access-c7sp8\") pod \"payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj\" (UID: \"3de3a976-5be0-4ba7-a869-4c58bb8fee0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj"
Oct 30 19:54:57.730001 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:57.729975    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de3a976-5be0-4ba7-a869-4c58bb8fee0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj\" (UID: \"3de3a976-5be0-4ba7-a869-4c58bb8fee0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj"
Oct 30 19:54:57.759169 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:54:57.756735    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11077-1db447\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="7a4b4eae-a145-4e7e-b255-14de8325d7a1" pod="opendatahub/test-curl-11077-1db447"
Oct 30 19:54:57.861052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:57.854252    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7sp8\" (UniqueName: \"kubernetes.io/projected/3de3a976-5be0-4ba7-a869-4c58bb8fee0f-kube-api-access-c7sp8\") pod \"payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj\" (UID: \"3de3a976-5be0-4ba7-a869-4c58bb8fee0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj"
Oct 30 19:54:57.861052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:57.854335    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de3a976-5be0-4ba7-a869-4c58bb8fee0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj\" (UID: \"3de3a976-5be0-4ba7-a869-4c58bb8fee0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj"
Oct 30 19:54:57.887052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:57.865472    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de3a976-5be0-4ba7-a869-4c58bb8fee0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj\" (UID: \"3de3a976-5be0-4ba7-a869-4c58bb8fee0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj"
Oct 30 19:54:57.981751 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:57.981720    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7sp8\" (UniqueName: \"kubernetes.io/projected/3de3a976-5be0-4ba7-a869-4c58bb8fee0f-kube-api-access-c7sp8\") pod \"payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj\" (UID: \"3de3a976-5be0-4ba7-a869-4c58bb8fee0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj"
Oct 30 19:54:58.220849 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:58.220765    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj"
Oct 30 19:54:58.382018 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:58.375299    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11077-1db447"
Oct 30 19:54:58.427596 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:54:58.400813    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11077-1db447\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="7a4b4eae-a145-4e7e-b255-14de8325d7a1" pod="opendatahub/test-curl-11077-1db447"
Oct 30 19:54:58.500847 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:58.500821    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a4b4eae-a145-4e7e-b255-14de8325d7a1-kube-api-access-lklsr\" (UniqueName: \"kubernetes.io/projected/7a4b4eae-a145-4e7e-b255-14de8325d7a1-kube-api-access-lklsr\") pod \"7a4b4eae-a145-4e7e-b255-14de8325d7a1\" (UID: \"7a4b4eae-a145-4e7e-b255-14de8325d7a1\") "
Oct 30 19:54:58.511022 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:58.510993    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a4b4eae-a145-4e7e-b255-14de8325d7a1-kube-api-access-lklsr" pod "7a4b4eae-a145-4e7e-b255-14de8325d7a1" (UID: "7a4b4eae-a145-4e7e-b255-14de8325d7a1"). InnerVolumeSpecName "kube-api-access-lklsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:58.602642 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:58.602610    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lklsr\" (UniqueName: \"kubernetes.io/projected/7a4b4eae-a145-4e7e-b255-14de8325d7a1-kube-api-access-lklsr\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:58.683138 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:58.683093    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj"]
Oct 30 19:54:58.685130 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:54:58.685107    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de3a976_5be0_4ba7_a869_4c58bb8fee0f.slice/crio-1998f26c53c463a6998e9de2e7af556dfd9e0d67b03097f8ab4586b51aeaf7d3 WatchSource:0}: Error finding container 1998f26c53c463a6998e9de2e7af556dfd9e0d67b03097f8ab4586b51aeaf7d3: Status 404 returned error can't find the container with id 1998f26c53c463a6998e9de2e7af556dfd9e0d67b03097f8ab4586b51aeaf7d3
Oct 30 19:54:58.884054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:58.877928    5408 generic.go:356] "Generic (PLEG): container finished" podID="7a4b4eae-a145-4e7e-b255-14de8325d7a1" containerID="a298f679c64134197431196131d4bdd4ba528c4dccf0f1366261d80c689db60c" exitCode=137
Oct 30 19:54:58.884054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:58.878716    5408 scope.go:122] "RemoveContainer" containerID="a298f679c64134197431196131d4bdd4ba528c4dccf0f1366261d80c689db60c"
Oct 30 19:54:58.884054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:58.878958    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11077-1db447"
Oct 30 19:54:58.924524 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:54:58.923506    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11077-1db447\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="7a4b4eae-a145-4e7e-b255-14de8325d7a1" pod="opendatahub/test-curl-11077-1db447"
Oct 30 19:54:58.926945 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:54:58.926653    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11077-1db447\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="7a4b4eae-a145-4e7e-b255-14de8325d7a1" pod="opendatahub/test-curl-11077-1db447"
Oct 30 19:54:58.938096 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:58.937604    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj" event={"ID":"3de3a976-5be0-4ba7-a869-4c58bb8fee0f","Type":"ContainerStarted","Data":"815d6f74856d8caaf929afcf384903252491ed8fc82d63d4bf3fe7d9e75acc36"}
Oct 30 19:54:58.938096 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:58.937675    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj" event={"ID":"3de3a976-5be0-4ba7-a869-4c58bb8fee0f","Type":"ContainerStarted","Data":"1998f26c53c463a6998e9de2e7af556dfd9e0d67b03097f8ab4586b51aeaf7d3"}
Oct 30 19:54:58.938810 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:58.938787    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj"
Oct 30 19:54:58.941631 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:54:58.941601    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11077-1db447\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="7a4b4eae-a145-4e7e-b255-14de8325d7a1" pod="opendatahub/test-curl-11077-1db447"
Oct 30 19:54:58.945671 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:58.945639    5408 scope.go:122] "RemoveContainer" containerID="a298f679c64134197431196131d4bdd4ba528c4dccf0f1366261d80c689db60c"
Oct 30 19:54:58.946908 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:54:58.946852    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a298f679c64134197431196131d4bdd4ba528c4dccf0f1366261d80c689db60c\": container with ID starting with a298f679c64134197431196131d4bdd4ba528c4dccf0f1366261d80c689db60c not found: ID does not exist" containerID="a298f679c64134197431196131d4bdd4ba528c4dccf0f1366261d80c689db60c"
Oct 30 19:54:58.947090 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:58.946921    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a298f679c64134197431196131d4bdd4ba528c4dccf0f1366261d80c689db60c"} err="failed to get container status \"a298f679c64134197431196131d4bdd4ba528c4dccf0f1366261d80c689db60c\": rpc error: code = NotFound desc = could not find container \"a298f679c64134197431196131d4bdd4ba528c4dccf0f1366261d80c689db60c\": container with ID starting with a298f679c64134197431196131d4bdd4ba528c4dccf0f1366261d80c689db60c not found: ID does not exist"
Oct 30 19:54:58.975107 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:58.975060    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj" podStartSLOduration=1.97501645 podStartE2EDuration="1.97501645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:58.972996987 +0000 UTC m=+1453.557266606" watchObservedRunningTime="2026-10-30 19:54:58.97501645 +0000 UTC m=+1453.559286066"
Oct 30 19:54:59.533250 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:54:59.533209    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a4b4eae-a145-4e7e-b255-14de8325d7a1" path="/var/lib/kubelet/pods/7a4b4eae-a145-4e7e-b255-14de8325d7a1/volumes"
Oct 30 19:55:01.286596 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:01.286566    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"]
Oct 30 19:55:01.292550 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:01.292525    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:01.303262 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:01.303231    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-maas-api-tls\") pod \"maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw\" (UID: \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:01.303383 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:01.303271    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-metrics-certs\") pod \"maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw\" (UID: \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:01.303451 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:01.303393    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-openshift-service-ca\") pod \"maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw\" (UID: \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:01.303550 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:01.303529    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-655c4\" (UniqueName: \"kubernetes.io/projected/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-kube-api-access-655c4\") pod \"maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw\" (UID: \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:01.319982 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:01.319947    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"]
Oct 30 19:55:01.327843 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:01.327818    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-df345a65-serving-cert\""
Oct 30 19:55:01.405484 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:01.405453    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-openshift-service-ca\") pod \"maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw\" (UID: \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:01.405661 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:01.405525    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-655c4\" (UniqueName: \"kubernetes.io/projected/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-kube-api-access-655c4\") pod \"maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw\" (UID: \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:01.405740 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:01.405698    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-maas-api-tls\") pod \"maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw\" (UID: \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:01.405740 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:01.405725    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-metrics-certs\") pod \"maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw\" (UID: \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:01.405907 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:55:01.405884    5408 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-serving-cert: secret "maas-api-e2e-ait-df345a65-serving-cert" not found
Oct 30 19:55:01.405987 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:55:01.405975    5408 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-maas-api-tls podName:3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef nodeName:}" failed. No retries permitted until 2026-10-30 19:55:01.905954546 +0000 UTC m=+1456.490224145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-maas-api-tls") pod "maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw" (UID: "3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef") : secret "maas-api-e2e-ait-df345a65-serving-cert" not found
Oct 30 19:55:01.406352 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:01.406325    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-openshift-service-ca\") pod \"maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw\" (UID: \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:01.408591 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:01.408559    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-metrics-certs\") pod \"maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw\" (UID: \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:01.421690 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:01.421656    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-655c4\" (UniqueName: \"kubernetes.io/projected/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-kube-api-access-655c4\") pod \"maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw\" (UID: \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:01.922752 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:01.922709    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-maas-api-tls\") pod \"maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw\" (UID: \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:01.925151 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:01.925109    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-maas-api-tls\") pod \"maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw\" (UID: \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:02.306150 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:02.228628    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:02.896972 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:02.896934    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"]
Oct 30 19:55:02.899067 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:55:02.899021    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef49356_5f15_4c7a_8699_ce6d2fd1e5ef.slice/crio-0d7c1ec55286e8d5a7c35758d9b3dae21eaa94879cbb1356eed9428c4be9b313 WatchSource:0}: Error finding container 0d7c1ec55286e8d5a7c35758d9b3dae21eaa94879cbb1356eed9428c4be9b313: Status 404 returned error can't find the container with id 0d7c1ec55286e8d5a7c35758d9b3dae21eaa94879cbb1356eed9428c4be9b313
Oct 30 19:55:02.971625 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:02.971592    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw" event={"ID":"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef","Type":"ContainerStarted","Data":"0d7c1ec55286e8d5a7c35758d9b3dae21eaa94879cbb1356eed9428c4be9b313"}
Oct 30 19:55:04.007393 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:04.007195    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw" event={"ID":"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef","Type":"ContainerStarted","Data":"7ada787634e7398fc213474f2954c0efa68c52e3257cd1caa22917a708b0314e"}
Oct 30 19:55:04.015192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:04.012949    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:04.039760 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:04.039714    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw" podStartSLOduration=2.625117317 podStartE2EDuration="3.039700919s" totalImagesPullingTime="414.583602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:04.037375337 +0000 UTC m=+1458.621645021" watchObservedRunningTime="2026-10-30 19:55:04.039700919 +0000 UTC m=+1458.623970536"
Oct 30 19:55:09.661107 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:55:09.660963    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:55:10.039662 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.039628    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:10.634568 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.634534    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"]
Oct 30 19:55:10.639802 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.639780    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.663122 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.663094    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"]
Oct 30 19:55:10.796053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.795974    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.796289 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.796172    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czf2b\" (UniqueName: \"kubernetes.io/projected/30b8551c-d7a1-4527-bd33-6f16c63ad713-kube-api-access-czf2b\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.796289 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.796253    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.796408 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.796347    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.796475 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.796464    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.796531 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.796499    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.796531 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.796523    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b8551c-d7a1-4527-bd33-6f16c63ad713-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.796635 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.796560    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.796635 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.796624    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.898352 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.898265    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.898513 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.898395    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.898513 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.898419    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.898513 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.898434    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b8551c-d7a1-4527-bd33-6f16c63ad713-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.898513 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.898454    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.898513 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.898506    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.898792 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.898554    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.898792 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.898699    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czf2b\" (UniqueName: \"kubernetes.io/projected/30b8551c-d7a1-4527-bd33-6f16c63ad713-kube-api-access-czf2b\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.898792 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.898782    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.898984 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.898956    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.899241 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.899218    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b8551c-d7a1-4527-bd33-6f16c63ad713-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.899351 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.899284    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.899415 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.899356    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.899506 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.899476    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.900730 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.900709    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.901352 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.901336    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.927341 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.927316    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.927495 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.927477    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czf2b\" (UniqueName: \"kubernetes.io/projected/30b8551c-d7a1-4527-bd33-6f16c63ad713-kube-api-access-czf2b\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.957479 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.957455    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:10.960377 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:10.960357    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj"
Oct 30 19:55:11.272934 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.272904    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"]
Oct 30 19:55:11.273321 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:55:11.273293    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b8551c_d7a1_4527_bd33_6f16c63ad713.slice/crio-db387da76aac9aa8135df48f122489c4060f7441b4a4ddc5ffa442158603fa9f WatchSource:0}: Error finding container db387da76aac9aa8135df48f122489c4060f7441b4a4ddc5ffa442158603fa9f: Status 404 returned error can't find the container with id db387da76aac9aa8135df48f122489c4060f7441b4a4ddc5ffa442158603fa9f
Oct 30 19:55:11.275928 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.275897    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:55:11.276011 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.275958    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:55:11.276011 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.276002    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:55:11.775449 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.775416    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"]
Oct 30 19:55:11.779772 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.779749    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.795993 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.795968    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"]
Oct 30 19:55:11.826413 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.826381    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad4e73d4-d56f-499f-ac82-01e963174b4d-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.826573 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.826431    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.826573 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.826503    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.826573 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.826551    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.826692 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.826602    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr498\" (UniqueName: \"kubernetes.io/projected/ad4e73d4-d56f-499f-ac82-01e963174b4d-kube-api-access-gr498\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.826692 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.826629    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.826692 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.826652    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.826692 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.826668    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.826823 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.826753    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.927662 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.927632    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.927810 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.927749    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr498\" (UniqueName: \"kubernetes.io/projected/ad4e73d4-d56f-499f-ac82-01e963174b4d-kube-api-access-gr498\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.927810 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.927798    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.928002 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.927837    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.928002 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.927872    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.928144 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.928000    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.928144 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.928058    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad4e73d4-d56f-499f-ac82-01e963174b4d-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.928144 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.928122    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.928144 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.928132    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.928351 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.928300    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.928351 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.928338    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.928454 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.928430    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.928454 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.928448    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.928684 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.928662    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad4e73d4-d56f-499f-ac82-01e963174b4d-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.930215 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.930196    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.930715 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.930685    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.943841 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.943808    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:11.945642 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:11.945618    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr498\" (UniqueName: \"kubernetes.io/projected/ad4e73d4-d56f-499f-ac82-01e963174b4d-kube-api-access-gr498\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:12.104624 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:12.104353    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:12.118741 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:12.118706    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" event={"ID":"30b8551c-d7a1-4527-bd33-6f16c63ad713","Type":"ContainerStarted","Data":"2185aaaf7d5bb4c253ea7068005c6a68475e80a78eea0c86bcd2121a1248a8f4"}
Oct 30 19:55:12.129210 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:12.129172    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" event={"ID":"30b8551c-d7a1-4527-bd33-6f16c63ad713","Type":"ContainerStarted","Data":"db387da76aac9aa8135df48f122489c4060f7441b4a4ddc5ffa442158603fa9f"}
Oct 30 19:55:12.180673 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:12.180610    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" podStartSLOduration=2.180589512 podStartE2EDuration="2.180589512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:12.164887071 +0000 UTC m=+1466.749156688" watchObservedRunningTime="2026-10-30 19:55:12.180589512 +0000 UTC m=+1466.764859130"
Oct 30 19:55:12.484263 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:12.484237    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"]
Oct 30 19:55:12.486389 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:55:12.486364    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4e73d4_d56f_499f_ac82_01e963174b4d.slice/crio-f5e79c21c75164f90941a2a5a85baddbe7984861f55f38981bb5c9420e87c711 WatchSource:0}: Error finding container f5e79c21c75164f90941a2a5a85baddbe7984861f55f38981bb5c9420e87c711: Status 404 returned error can't find the container with id f5e79c21c75164f90941a2a5a85baddbe7984861f55f38981bb5c9420e87c711
Oct 30 19:55:12.489231 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:12.489198    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:55:12.489306 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:12.489276    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:55:12.489342 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:12.489308    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:55:12.958520 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:12.958491    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:12.960113 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:12.960092    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:55:12.960206 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:12.960139    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:55:13.046161 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.046106    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"]
Oct 30 19:55:13.051575 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.051555    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"]
Oct 30 19:55:13.051737 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.051668    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"
Oct 30 19:55:13.058763 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.058743    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-d07cb404-serving-cert\""
Oct 30 19:55:13.070116 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.070090    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42281754-675b-4589-8ce6-80ec15bda7a3-metrics-certs\") pod \"maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69\" (UID: \"42281754-675b-4589-8ce6-80ec15bda7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"
Oct 30 19:55:13.070223 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.070155    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42281754-675b-4589-8ce6-80ec15bda7a3-maas-api-tls\") pod \"maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69\" (UID: \"42281754-675b-4589-8ce6-80ec15bda7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"
Oct 30 19:55:13.070425 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.070405    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh586\" (UniqueName: \"kubernetes.io/projected/42281754-675b-4589-8ce6-80ec15bda7a3-kube-api-access-fh586\") pod \"maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69\" (UID: \"42281754-675b-4589-8ce6-80ec15bda7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"
Oct 30 19:55:13.070476 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.070462    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42281754-675b-4589-8ce6-80ec15bda7a3-openshift-service-ca\") pod \"maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69\" (UID: \"42281754-675b-4589-8ce6-80ec15bda7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"
Oct 30 19:55:13.134328 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.134290    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs" event={"ID":"ad4e73d4-d56f-499f-ac82-01e963174b4d","Type":"ContainerStarted","Data":"5e9066a512fc0cac12e38031ad1e2f13d09b1cc6575a432eaaf83a282af39c11"}
Oct 30 19:55:13.134505 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.134357    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs" event={"ID":"ad4e73d4-d56f-499f-ac82-01e963174b4d","Type":"ContainerStarted","Data":"f5e79c21c75164f90941a2a5a85baddbe7984861f55f38981bb5c9420e87c711"}
Oct 30 19:55:13.171648 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.171610    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh586\" (UniqueName: \"kubernetes.io/projected/42281754-675b-4589-8ce6-80ec15bda7a3-kube-api-access-fh586\") pod \"maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69\" (UID: \"42281754-675b-4589-8ce6-80ec15bda7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"
Oct 30 19:55:13.171859 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.171680    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42281754-675b-4589-8ce6-80ec15bda7a3-openshift-service-ca\") pod \"maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69\" (UID: \"42281754-675b-4589-8ce6-80ec15bda7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"
Oct 30 19:55:13.171939 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.171883    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42281754-675b-4589-8ce6-80ec15bda7a3-metrics-certs\") pod \"maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69\" (UID: \"42281754-675b-4589-8ce6-80ec15bda7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"
Oct 30 19:55:13.172008 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.171951    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42281754-675b-4589-8ce6-80ec15bda7a3-maas-api-tls\") pod \"maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69\" (UID: \"42281754-675b-4589-8ce6-80ec15bda7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"
Oct 30 19:55:13.172461 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.172433    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42281754-675b-4589-8ce6-80ec15bda7a3-openshift-service-ca\") pod \"maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69\" (UID: \"42281754-675b-4589-8ce6-80ec15bda7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"
Oct 30 19:55:13.174313 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.174285    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42281754-675b-4589-8ce6-80ec15bda7a3-metrics-certs\") pod \"maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69\" (UID: \"42281754-675b-4589-8ce6-80ec15bda7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"
Oct 30 19:55:13.174417 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.174355    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42281754-675b-4589-8ce6-80ec15bda7a3-maas-api-tls\") pod \"maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69\" (UID: \"42281754-675b-4589-8ce6-80ec15bda7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"
Oct 30 19:55:13.239098 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.233828    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh586\" (UniqueName: \"kubernetes.io/projected/42281754-675b-4589-8ce6-80ec15bda7a3-kube-api-access-fh586\") pod \"maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69\" (UID: \"42281754-675b-4589-8ce6-80ec15bda7a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"
Oct 30 19:55:13.375426 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.375389    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"
Oct 30 19:55:13.885111 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.882863    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs" podStartSLOduration=2.88283999 podStartE2EDuration="2.88283999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:13.198285178 +0000 UTC m=+1467.782554796" watchObservedRunningTime="2026-10-30 19:55:13.88283999 +0000 UTC m=+1468.467109614"
Oct 30 19:55:13.885111 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.884762    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"]
Oct 30 19:55:13.961083 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.960448    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:55:13.961083 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:13.960506    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:55:14.120070 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:14.117387    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:14.196306 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:14.196177    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:55:14.196638 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:14.196587    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs" podUID="ad4e73d4-d56f-499f-ac82-01e963174b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:55:14.495368 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:14.495303    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69" event={"ID":"42281754-675b-4589-8ce6-80ec15bda7a3","Type":"ContainerStarted","Data":"1e6779117c5f0d23f6b64cd7188eb797bad0d8bf43e5acb58564007e4cefcb3e"}
Oct 30 19:55:14.977105 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:14.977069    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:55:14.977677 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:14.977642    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:55:15.165065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:15.152267    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:55:15.165065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:15.152332    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs" podUID="ad4e73d4-d56f-499f-ac82-01e963174b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:55:15.799901 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:15.799492    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69" event={"ID":"42281754-675b-4589-8ce6-80ec15bda7a3","Type":"ContainerStarted","Data":"92f9c3280a4a6269420abd8058dce18bf3d73c63010e782830df396763272968"}
Oct 30 19:55:15.799901 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:15.799565    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"
Oct 30 19:55:15.842055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:15.839018    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69" podStartSLOduration=3.002768816 podStartE2EDuration="3.838993964s" totalImagesPullingTime="836.225148ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:15.816437782 +0000 UTC m=+1470.400707400" watchObservedRunningTime="2026-10-30 19:55:15.838993964 +0000 UTC m=+1470.423263583"
Oct 30 19:55:15.962490 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:15.962457    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:55:15.962783 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:15.962755    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:55:16.181120 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:16.177131    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:55:16.181120 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:16.177207    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs" podUID="ad4e73d4-d56f-499f-ac82-01e963174b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:55:16.518052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:16.517420    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:16.773547 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:16.773448    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:16.791613 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:16.791172    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:55:16.974051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:16.960988    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:55:16.974051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:16.961070    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:55:17.083836 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:17.054075    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:17.111720 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:17.056218    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:55:17.146850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:17.075619    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:17.147215 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:17.084125    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:17.192729 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:17.192685    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:55:17.194162 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:17.194133    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:55:17.201160 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:17.201117    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs" podUID="ad4e73d4-d56f-499f-ac82-01e963174b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:55:17.255158 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:17.255111    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:55:17.264252 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:17.264198    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:17.663457 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:17.663411    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:17.710474 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:17.710421    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:17.782000 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:17.781952    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:17.829264 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:17.817152    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:55:17.829264 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:17.799148    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:17.829264 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:17.818764    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:55:17.982105 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:17.982002    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:55:17.982357 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:17.982327    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:55:18.160053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.141151    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:18.182992 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.182950    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:55:18.291663 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.209583    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:18.291663 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.231857    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:55:18.291663 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.285552    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs" podUID="ad4e73d4-d56f-499f-ac82-01e963174b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:55:18.291663 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.231939    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:55:18.291663 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.286266    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:55:18.291663 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.286333    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:55:18.291663 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.231907    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:18.291663 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.268320    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:18.291663 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.274159    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:18.291663 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.287726    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:55:18.291663 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.287776    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:55:18.291663 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.287810    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:55:18.336065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.335938    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:18.550106 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.523524    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:18.836791 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.836669    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:55:18.837567 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.837537    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:55:18.973043 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.972989    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:55:18.973335 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:18.973304    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:55:19.118208 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.118097    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:55:19.118488 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.118442    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs" podUID="ad4e73d4-d56f-499f-ac82-01e963174b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:55:19.205829 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.204545    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:19.326061 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.320645    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:19.326061 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.322043    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:55:19.326061 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.322674    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:55:19.326061 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.323784    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 19:55:19.326061 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.324800    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:55:19.562645 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.562597    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:19.822484 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.822391    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:55:19.891930 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.873358    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc" podUID="084af46f-59a7-4518-8938-2b4e09979c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:19.891930 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.872345    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:19.891930 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.872103    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:55:19.935056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.925601    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:55:19.935056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.925969    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:55:19.971052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.967485    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:55:19.971052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.968432    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:55:19.971052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.968480    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:55:19.984160 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:19.975823    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:55:20.170211 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:20.159833    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:55:20.170211 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:20.159894    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs" podUID="ad4e73d4-d56f-499f-ac82-01e963174b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:55:20.351059 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:20.347192    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:20.351059 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:20.347688    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 19:55:20.351059 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:20.348018    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:20.449336 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:20.439121    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded"
Oct 30 19:55:20.634609 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:20.634569    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:55:20.875181 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:20.875130    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:55:20.974242 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:20.974183    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:55:20.977766 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:20.977737    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:55:20.978002 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:20.977977    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:55:21.107598 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:21.107563    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:55:21.107915 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:21.107871    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs" podUID="ad4e73d4-d56f-499f-ac82-01e963174b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:55:21.329347 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:21.329304    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:21.330367 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:21.329622    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:21.356798 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:21.356759    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:21.434061 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:21.389937    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:21.434061 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:21.389979    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:21.564153 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:21.564108    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:21.658572 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:21.658482    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:21.889951 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:21.889911    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:55:21.890325 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:21.890298    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq" podUID="688f0cb2-8d4a-47ab-ab79-fb1a54109e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:21.890706 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:21.890239    5408 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:55:21.890855 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:21.890834    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84" podUID="a8c73d1b-a864-4534-944a-a543d04a2c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:21.935273 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:21.935169    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:22.046050 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:22.016550    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:22.047725 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:22.024662    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:55:22.074055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:22.047914    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:55:22.108260 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:22.027558    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:55:22.131426 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:22.131393    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:55:22.137134 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:22.137097    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs" podUID="ad4e73d4-d56f-499f-ac82-01e963174b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:55:22.374060 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:22.366289    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:22.454966 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:22.407786    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:22.456468 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:22.407848    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded"
Oct 30 19:55:22.479810 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:22.435025    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:55:22.481156 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:22.435089    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 19:55:22.494562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:22.435110    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:55:22.494562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:22.435604    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"
Oct 30 19:55:22.494562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:22.482881    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:55:22.494562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:22.482927    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:55:22.677383 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:22.676176    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:22.978329 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:22.958385    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:55:23.004078 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:22.983073    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:55:23.079171 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.079117    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:55:23.356053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.332109    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:55:23.357435 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.357399    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:55:23.385601 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.385571    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:55:23.484111 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.457105    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:23.502243 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.502196    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:23.503719 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.503059    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:23.503719 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.503410    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:23.522163 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.454024    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:55:23.522420 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.522393    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs" podUID="ad4e73d4-d56f-499f-ac82-01e963174b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:55:23.562985 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.550292    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:55:23.583560 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.550386    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:23.583560 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.560129    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:55:23.787412 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.761117    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:23.965501 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.965459    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:55:23.967145 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.967113    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:55:23.968108 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.967757    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:55:23.996392 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.987457    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:55:23.996392 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:23.992486    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:55:24.005411 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:24.005386    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:55:24.112815 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:24.109237    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:55:24.112815 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:24.109303    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs" podUID="ad4e73d4-d56f-499f-ac82-01e963174b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:55:24.178051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:24.178011    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:55:24.192060 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:24.189334    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:55:24.198085 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:24.198063    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:55:24.198725 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:24.198704    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:55:24.958355 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:24.958314    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:55:24.958830 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:24.958395    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:55:25.105859 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:25.105824    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:55:25.106080 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:25.105885    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs" podUID="ad4e73d4-d56f-499f-ac82-01e963174b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:55:25.958602 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:25.958561    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:26.958078 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:26.957509    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:26.959628 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:26.959608    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:26.965738 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:26.965709    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 19:55:27.000423 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:27.000377    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:55:27.000589 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:27.000450    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:27.175986 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:27.175957    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:27.177140 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:27.177119    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 19:55:39.377592 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:39.377554    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"]
Oct 30 19:55:39.378355 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:39.378322    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw" podUID="3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef" containerName="maas-api" containerID="cri-o://7ada787634e7398fc213474f2954c0efa68c52e3257cd1caa22917a708b0314e" gracePeriod=30
Oct 30 19:55:39.770844 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:39.770810    5408 generic.go:356] "Generic (PLEG): container finished" podID="3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef" containerID="7ada787634e7398fc213474f2954c0efa68c52e3257cd1caa22917a708b0314e" exitCode=0
Oct 30 19:55:39.771187 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:39.771163    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw" event={"ID":"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef","Type":"ContainerDied","Data":"7ada787634e7398fc213474f2954c0efa68c52e3257cd1caa22917a708b0314e"}
Oct 30 19:55:39.984775 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:39.984740    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:40.029525 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:40.024917    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-kube-api-access-655c4\" (UniqueName: \"kubernetes.io/projected/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-kube-api-access-655c4\") pod \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\" (UID: \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\") "
Oct 30 19:55:40.029807 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:40.029782    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-maas-api-tls\") pod \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\" (UID: \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\") "
Oct 30 19:55:40.030166 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:40.030144    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-openshift-service-ca\") pod \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\" (UID: \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\") "
Oct 30 19:55:40.030375 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:40.030357    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-metrics-certs\") pod \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\" (UID: \"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef\") "
Oct 30 19:55:40.032046 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:40.032008    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-openshift-service-ca" pod "3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef" (UID: "3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:40.040136 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:55:40.039960    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:55:40.044741 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:40.044687    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-maas-api-tls" pod "3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef" (UID: "3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:40.046239 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:40.045319    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-kube-api-access-655c4" pod "3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef" (UID: "3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef"). InnerVolumeSpecName "kube-api-access-655c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:40.051355 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:40.051324    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-metrics-certs" pod "3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef" (UID: "3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:40.132932 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:40.132893    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:40.132932 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:40.132925    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:40.132932 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:40.132939    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-655c4\" (UniqueName: \"kubernetes.io/projected/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-kube-api-access-655c4\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:40.133200 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:40.132951    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:40.830647 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:40.830419    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw" event={"ID":"3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef","Type":"ContainerDied","Data":"0d7c1ec55286e8d5a7c35758d9b3dae21eaa94879cbb1356eed9428c4be9b313"}
Oct 30 19:55:40.830647 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:40.830498    5408 scope.go:122] "RemoveContainer" containerID="7ada787634e7398fc213474f2954c0efa68c52e3257cd1caa22917a708b0314e"
Oct 30 19:55:40.831208 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:40.830666    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"
Oct 30 19:55:40.892021 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:40.891992    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"]
Oct 30 19:55:40.897784 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:40.897760    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df345a65-7759dbd9c9-hrmgw"]
Oct 30 19:55:41.532345 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:41.532315    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef" path="/var/lib/kubelet/pods/3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef/volumes"
Oct 30 19:55:46.464127 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.464049    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh"]
Oct 30 19:55:46.464954 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.464927    5408 cpu_manager.go:414] "containerMap: removing container" podUID="3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef" containerName="maas-api"
Oct 30 19:55:46.464954 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.464952    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef" containerName="maas-api"
Oct 30 19:55:46.465199 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.465184    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef49356-5f15-4c7a-8699-ce6d2fd1e5ef" containerName="maas-api"
Oct 30 19:55:46.470747 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.470728    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf"]
Oct 30 19:55:46.470866 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.470850    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh"
Oct 30 19:55:46.474899 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.474879    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf"
Oct 30 19:55:46.479996 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.479977    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b69bf89d\""
Oct 30 19:55:46.486777 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.486753    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b69bf89d-dockercfg-p4x48\""
Oct 30 19:55:46.491434 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.491157    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh"]
Oct 30 19:55:46.495416 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.495394    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf"]
Oct 30 19:55:46.570731 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.570688    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ea6a85-2031-4c7a-ad94-9055c6a8f10a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh\" (UID: \"06ea6a85-2031-4c7a-ad94-9055c6a8f10a\") " pod="openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh"
Oct 30 19:55:46.570945 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.570922    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lbc6\" (UniqueName: \"kubernetes.io/projected/a7102374-9148-4b5d-b21a-6b5b00d5022a-kube-api-access-2lbc6\") pod \"payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf\" (UID: \"a7102374-9148-4b5d-b21a-6b5b00d5022a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf"
Oct 30 19:55:46.571005 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.570983    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7102374-9148-4b5d-b21a-6b5b00d5022a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf\" (UID: \"a7102374-9148-4b5d-b21a-6b5b00d5022a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf"
Oct 30 19:55:46.571092 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.571072    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpmtl\" (UniqueName: \"kubernetes.io/projected/06ea6a85-2031-4c7a-ad94-9055c6a8f10a-kube-api-access-fpmtl\") pod \"payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh\" (UID: \"06ea6a85-2031-4c7a-ad94-9055c6a8f10a\") " pod="openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh"
Oct 30 19:55:46.681254 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.673355    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lbc6\" (UniqueName: \"kubernetes.io/projected/a7102374-9148-4b5d-b21a-6b5b00d5022a-kube-api-access-2lbc6\") pod \"payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf\" (UID: \"a7102374-9148-4b5d-b21a-6b5b00d5022a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf"
Oct 30 19:55:46.681254 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.673457    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7102374-9148-4b5d-b21a-6b5b00d5022a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf\" (UID: \"a7102374-9148-4b5d-b21a-6b5b00d5022a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf"
Oct 30 19:55:46.681254 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.673493    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpmtl\" (UniqueName: \"kubernetes.io/projected/06ea6a85-2031-4c7a-ad94-9055c6a8f10a-kube-api-access-fpmtl\") pod \"payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh\" (UID: \"06ea6a85-2031-4c7a-ad94-9055c6a8f10a\") " pod="openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh"
Oct 30 19:55:46.681254 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.673689    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ea6a85-2031-4c7a-ad94-9055c6a8f10a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh\" (UID: \"06ea6a85-2031-4c7a-ad94-9055c6a8f10a\") " pod="openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh"
Oct 30 19:55:46.681254 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.674734    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ea6a85-2031-4c7a-ad94-9055c6a8f10a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh\" (UID: \"06ea6a85-2031-4c7a-ad94-9055c6a8f10a\") " pod="openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh"
Oct 30 19:55:46.681254 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.675883    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7102374-9148-4b5d-b21a-6b5b00d5022a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf\" (UID: \"a7102374-9148-4b5d-b21a-6b5b00d5022a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf"
Oct 30 19:55:46.713055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.712719    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lbc6\" (UniqueName: \"kubernetes.io/projected/a7102374-9148-4b5d-b21a-6b5b00d5022a-kube-api-access-2lbc6\") pod \"payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf\" (UID: \"a7102374-9148-4b5d-b21a-6b5b00d5022a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf"
Oct 30 19:55:46.744119 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.742822    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpmtl\" (UniqueName: \"kubernetes.io/projected/06ea6a85-2031-4c7a-ad94-9055c6a8f10a-kube-api-access-fpmtl\") pod \"payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh\" (UID: \"06ea6a85-2031-4c7a-ad94-9055c6a8f10a\") " pod="openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh"
Oct 30 19:55:46.789066 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.788005    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh"
Oct 30 19:55:46.807051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:46.805540    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf"
Oct 30 19:55:47.288499 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:55:47.288469    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ea6a85_2031_4c7a_ad94_9055c6a8f10a.slice/crio-4248a4194fe09c81d43db656dd1d2e80562b4df34990fa07b48796f6c9b9b26f WatchSource:0}: Error finding container 4248a4194fe09c81d43db656dd1d2e80562b4df34990fa07b48796f6c9b9b26f: Status 404 returned error can't find the container with id 4248a4194fe09c81d43db656dd1d2e80562b4df34990fa07b48796f6c9b9b26f
Oct 30 19:55:47.291720 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:47.291697    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh"]
Oct 30 19:55:47.296065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:47.296041    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf"]
Oct 30 19:55:47.557718 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:47.557627    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj"]
Oct 30 19:55:47.999067 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:47.996737    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf" event={"ID":"a7102374-9148-4b5d-b21a-6b5b00d5022a","Type":"ContainerStarted","Data":"19de52b437b211a660bfd1379014f900e7dcf727a8342207302bc379ed861d37"}
Oct 30 19:55:47.999067 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:47.996820    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf" event={"ID":"a7102374-9148-4b5d-b21a-6b5b00d5022a","Type":"ContainerStarted","Data":"b244ddb9a90e2abb4412f49ba1baa43bc9389b3045aa4601556f7869d72e7cd5"}
Oct 30 19:55:47.999067 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:47.998275    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf"
Oct 30 19:55:48.009054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:48.000550    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh" event={"ID":"06ea6a85-2031-4c7a-ad94-9055c6a8f10a","Type":"ContainerStarted","Data":"822dfa1d5e5e2b79cc706f6b697a4691cb6a635137961c7e206a443960dca739"}
Oct 30 19:55:48.009054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:48.000601    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh" event={"ID":"06ea6a85-2031-4c7a-ad94-9055c6a8f10a","Type":"ContainerStarted","Data":"4248a4194fe09c81d43db656dd1d2e80562b4df34990fa07b48796f6c9b9b26f"}
Oct 30 19:55:48.009054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:48.005542    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh"
Oct 30 19:55:48.052502 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:48.048420    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf" podStartSLOduration=2.048397088 podStartE2EDuration="2.048397088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:48.028890145 +0000 UTC m=+1502.613159763" watchObservedRunningTime="2026-10-30 19:55:48.048397088 +0000 UTC m=+1502.632666719"
Oct 30 19:55:51.890195 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:51.890146    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:52.188294 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:52.188165    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:52.288186 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:52.288140    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:52.328192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:52.328142    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:52.487219 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:52.487114    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:52.488485 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:52.488456    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:52.568054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:52.564099    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj" podUID="3de3a976-5be0-4ba7-a869-4c58bb8fee0f" containerName="payload-pre-processing" containerID="cri-o://815d6f74856d8caaf929afcf384903252491ed8fc82d63d4bf3fe7d9e75acc36" gracePeriod=25
Oct 30 19:55:52.586425 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:52.579133    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:52.747133 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:52.746338    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:52.853237 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:52.853175    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:52.992259 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:52.904401    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:52.992259 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:52.938156    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:55:53.003274 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:52.994826    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh" podStartSLOduration=6.994809292 podStartE2EDuration="6.994809292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:48.082132557 +0000 UTC m=+1502.666402187" watchObservedRunningTime="2026-10-30 19:55:52.994809292 +0000 UTC m=+1507.579078910"
Oct 30 19:55:53.237894 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:52.995008    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.237894 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:53.196309    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.237894 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:53.215716    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:55:53.328654 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:53.328563    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.574600 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:53.504111    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.574600 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:53.505758    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:55:53.587536 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:53.584221    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.767471 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:53.757160    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.839109 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:53.836942    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:55:53.859363 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:53.859315    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.894826 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:53.889380    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:55:53.895244 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:53.895215    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:55:53.896663 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:53.896645    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:55:53.897599 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:53.897583    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:55:53.907498 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:53.907455    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:54.031327 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:54.031136    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:54.031838 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:54.031816    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:55:54.210118 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:54.197766    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:54.540541 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:54.540494    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:54.566198 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:54.566158    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:55:54.584605 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:54.584566    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:55:54.585825 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:54.585791    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:54.613852 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:54.613410    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:54.645729 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:54.645441    5408 generic.go:356] "Generic (PLEG): container finished" podID="3de3a976-5be0-4ba7-a869-4c58bb8fee0f" containerID="815d6f74856d8caaf929afcf384903252491ed8fc82d63d4bf3fe7d9e75acc36" exitCode=0
Oct 30 19:55:54.645729 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:54.645645    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj" event={"ID":"3de3a976-5be0-4ba7-a869-4c58bb8fee0f","Type":"ContainerDied","Data":"815d6f74856d8caaf929afcf384903252491ed8fc82d63d4bf3fe7d9e75acc36"}
Oct 30 19:55:54.759371 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:54.759322    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:54.890241 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:54.890130    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:55:54.908797 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:54.908756    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:55.039064 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:55.033145    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:55.243182 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:55.243134    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:55.563304 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:55.563199    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 19:55:55.564138 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:55.564113    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:55:55.614106 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:55.608728    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:55.625071 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:55.615834    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:55.625071 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:55.616231    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:55:55.818074 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:55.817956    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:56.051838 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.051788    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:55:56.312641 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.312602    5408 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:55:56.312992 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.312961    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" podUID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:56.347085 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.344268    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:55:56.371061 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.369523    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:56.420054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.404496    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:55:56.446054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.442308    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:55:56.446054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.442414    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn" podUID="e8795c70-cfa2-46d3-85a9-3af34e81144d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:56.568663 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.568556    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:56.627170 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.607488    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:55:56.627170 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.607551    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp" podUID="0a0737c2-db49-4cb0-b815-0b2c9560119e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:56.627170 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.611432    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:56.627170 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.617122    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:56.675300 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.675264    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:55:56.815075 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:55:56.804309    5408 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.233s"
Oct 30 19:55:56.853297 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.853208    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:55:56.878175 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.876436    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:55:56.909211 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.907991    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:55:56.960648 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.951894    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj"
Oct 30 19:55:56.986617 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.986582    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de3a976-5be0-4ba7-a869-4c58bb8fee0f-kube-api-access-c7sp8\" (UniqueName: \"kubernetes.io/projected/3de3a976-5be0-4ba7-a869-4c58bb8fee0f-kube-api-access-c7sp8\") pod \"3de3a976-5be0-4ba7-a869-4c58bb8fee0f\" (UID: \"3de3a976-5be0-4ba7-a869-4c58bb8fee0f\") "
Oct 30 19:55:56.987309 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.987285    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de3a976-5be0-4ba7-a869-4c58bb8fee0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de3a976-5be0-4ba7-a869-4c58bb8fee0f-plugins-config-volume\") pod \"3de3a976-5be0-4ba7-a869-4c58bb8fee0f\" (UID: \"3de3a976-5be0-4ba7-a869-4c58bb8fee0f\") "
Oct 30 19:55:56.987919 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.987815    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de3a976-5be0-4ba7-a869-4c58bb8fee0f-plugins-config-volume" pod "3de3a976-5be0-4ba7-a869-4c58bb8fee0f" (UID: "3de3a976-5be0-4ba7-a869-4c58bb8fee0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:56.993826 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.993808    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de3a976-5be0-4ba7-a869-4c58bb8fee0f-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:57.007664 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:56.999745    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de3a976-5be0-4ba7-a869-4c58bb8fee0f-kube-api-access-c7sp8" pod "3de3a976-5be0-4ba7-a869-4c58bb8fee0f" (UID: "3de3a976-5be0-4ba7-a869-4c58bb8fee0f"). InnerVolumeSpecName "kube-api-access-c7sp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:57.096052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:57.095503    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7sp8\" (UniqueName: \"kubernetes.io/projected/3de3a976-5be0-4ba7-a869-4c58bb8fee0f-kube-api-access-c7sp8\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:57.404863 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:57.404804    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:57.436697 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:57.436120    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:57.491473 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:57.487067    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:57.491473 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:57.490597    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj" event={"ID":"3de3a976-5be0-4ba7-a869-4c58bb8fee0f","Type":"ContainerDied","Data":"1998f26c53c463a6998e9de2e7af556dfd9e0d67b03097f8ab4586b51aeaf7d3"}
Oct 30 19:55:57.491473 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:57.490672    5408 scope.go:122] "RemoveContainer" containerID="815d6f74856d8caaf929afcf384903252491ed8fc82d63d4bf3fe7d9e75acc36"
Oct 30 19:55:57.491473 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:57.490878    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj"
Oct 30 19:55:57.570953 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:57.570097    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:57.582601 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:57.582541    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj"]
Oct 30 19:55:57.586801 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:57.586777    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea210528-6dcbff56c9-g8bzj"]
Oct 30 19:55:57.607726 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:57.607706    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:55:57.627400 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:57.627365    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:55:57.628967 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:57.628547    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:57.662049 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:57.658499    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:55:57.681051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:57.679121    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:57.743065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:57.738437    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:57.804054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:57.784507    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:57.885395 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:57.885364    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 19:55:58.142054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:58.134312    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 19:55:58.164691 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:58.154452    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 19:55:58.247525 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:58.247498    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 19:55:58.462019 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:58.461923    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:58.600376 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:58.600346    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:55:59.535191 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:55:59.535154    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3de3a976-5be0-4ba7-a869-4c58bb8fee0f" path="/var/lib/kubelet/pods/3de3a976-5be0-4ba7-a869-4c58bb8fee0f/volumes"
Oct 30 19:56:00.033073 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:00.033003    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh"
Oct 30 19:56:00.033313 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:00.033193    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf"
Oct 30 19:56:02.905938 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:02.905905    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"]
Oct 30 19:56:02.906860 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:02.906835    5408 cpu_manager.go:414] "containerMap: removing container" podUID="3de3a976-5be0-4ba7-a869-4c58bb8fee0f" containerName="payload-pre-processing"
Oct 30 19:56:02.907024 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:02.907000    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de3a976-5be0-4ba7-a869-4c58bb8fee0f" containerName="payload-pre-processing"
Oct 30 19:56:02.907255 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:02.907237    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de3a976-5be0-4ba7-a869-4c58bb8fee0f" containerName="payload-pre-processing"
Oct 30 19:56:02.912469 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:02.912449    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"
Oct 30 19:56:02.942366 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:02.942339    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b69bf89d-serving-cert\""
Oct 30 19:56:02.945086 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:02.945062    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"]
Oct 30 19:56:03.038561 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:03.038531    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba7faec-f569-467c-935b-1d66fb69f44d-metrics-certs\") pod \"maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m\" (UID: \"8ba7faec-f569-467c-935b-1d66fb69f44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"
Oct 30 19:56:03.038712 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:03.038660    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwrhw\" (UniqueName: \"kubernetes.io/projected/8ba7faec-f569-467c-935b-1d66fb69f44d-kube-api-access-xwrhw\") pod \"maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m\" (UID: \"8ba7faec-f569-467c-935b-1d66fb69f44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"
Oct 30 19:56:03.038760 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:03.038751    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba7faec-f569-467c-935b-1d66fb69f44d-maas-api-tls\") pod \"maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m\" (UID: \"8ba7faec-f569-467c-935b-1d66fb69f44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"
Oct 30 19:56:03.038900 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:03.038879    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba7faec-f569-467c-935b-1d66fb69f44d-openshift-service-ca\") pod \"maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m\" (UID: \"8ba7faec-f569-467c-935b-1d66fb69f44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"
Oct 30 19:56:03.152409 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:03.140750    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba7faec-f569-467c-935b-1d66fb69f44d-metrics-certs\") pod \"maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m\" (UID: \"8ba7faec-f569-467c-935b-1d66fb69f44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"
Oct 30 19:56:03.152409 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:03.140901    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwrhw\" (UniqueName: \"kubernetes.io/projected/8ba7faec-f569-467c-935b-1d66fb69f44d-kube-api-access-xwrhw\") pod \"maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m\" (UID: \"8ba7faec-f569-467c-935b-1d66fb69f44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"
Oct 30 19:56:03.152409 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:03.141052    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba7faec-f569-467c-935b-1d66fb69f44d-maas-api-tls\") pod \"maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m\" (UID: \"8ba7faec-f569-467c-935b-1d66fb69f44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"
Oct 30 19:56:03.152409 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:03.141116    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba7faec-f569-467c-935b-1d66fb69f44d-openshift-service-ca\") pod \"maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m\" (UID: \"8ba7faec-f569-467c-935b-1d66fb69f44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"
Oct 30 19:56:03.152409 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:03.142306    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba7faec-f569-467c-935b-1d66fb69f44d-openshift-service-ca\") pod \"maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m\" (UID: \"8ba7faec-f569-467c-935b-1d66fb69f44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"
Oct 30 19:56:03.157086 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:03.154796    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba7faec-f569-467c-935b-1d66fb69f44d-metrics-certs\") pod \"maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m\" (UID: \"8ba7faec-f569-467c-935b-1d66fb69f44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"
Oct 30 19:56:03.161046 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:03.159732    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba7faec-f569-467c-935b-1d66fb69f44d-maas-api-tls\") pod \"maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m\" (UID: \"8ba7faec-f569-467c-935b-1d66fb69f44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"
Oct 30 19:56:03.175049 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:03.173864    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwrhw\" (UniqueName: \"kubernetes.io/projected/8ba7faec-f569-467c-935b-1d66fb69f44d-kube-api-access-xwrhw\") pod \"maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m\" (UID: \"8ba7faec-f569-467c-935b-1d66fb69f44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"
Oct 30 19:56:03.230080 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:03.227454    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"
Oct 30 19:56:03.542785 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:03.542760    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"]
Oct 30 19:56:03.546257 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:56:03.546231    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba7faec_f569_467c_935b_1d66fb69f44d.slice/crio-fc5884b924d01cd56938db6dfde20120f26488226b535d325c1100a92769612f WatchSource:0}: Error finding container fc5884b924d01cd56938db6dfde20120f26488226b535d325c1100a92769612f: Status 404 returned error can't find the container with id fc5884b924d01cd56938db6dfde20120f26488226b535d325c1100a92769612f
Oct 30 19:56:03.720209 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:03.720168    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m" event={"ID":"8ba7faec-f569-467c-935b-1d66fb69f44d","Type":"ContainerStarted","Data":"fc5884b924d01cd56938db6dfde20120f26488226b535d325c1100a92769612f"}
Oct 30 19:56:04.728336 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:04.728301    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m" event={"ID":"8ba7faec-f569-467c-935b-1d66fb69f44d","Type":"ContainerStarted","Data":"99b3aae0a5889c799bb4e6c40a0c31fdeea29f5030cf99b9e8c62dc4c56cb727"}
Oct 30 19:56:04.728784 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:04.728361    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"
Oct 30 19:56:04.789301 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:04.789255    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m" podStartSLOduration=2.3318557220000002 podStartE2EDuration="2.789232723s" totalImagesPullingTime="457.377001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:04.787281916 +0000 UTC m=+1519.371551536" watchObservedRunningTime="2026-10-30 19:56:04.789232723 +0000 UTC m=+1519.373502340"
Oct 30 19:56:05.021427 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.021394    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p"]
Oct 30 19:56:05.025729 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.025711    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p"
Oct 30 19:56:05.037384 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.037364    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-a8de3b\""
Oct 30 19:56:05.045587 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.045566    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p"]
Oct 30 19:56:05.082929 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.082903    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4"]
Oct 30 19:56:05.087436 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.087417    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4"
Oct 30 19:56:05.104280 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.104258    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-a8de3b-dockercfg-wzdlc\""
Oct 30 19:56:05.104395 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.104358    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4"]
Oct 30 19:56:05.208072 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.200641    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twrxp\" (UniqueName: \"kubernetes.io/projected/39539848-97ad-40d8-8082-d27a15143388-kube-api-access-twrxp\") pod \"payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p\" (UID: \"39539848-97ad-40d8-8082-d27a15143388\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p"
Oct 30 19:56:05.208072 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.200846    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjns7\" (UniqueName: \"kubernetes.io/projected/686b83f0-b8a4-4210-a66d-e92c30e7fc8c-kube-api-access-mjns7\") pod \"payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4\" (UID: \"686b83f0-b8a4-4210-a66d-e92c30e7fc8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4"
Oct 30 19:56:05.208072 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.200896    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39539848-97ad-40d8-8082-d27a15143388-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p\" (UID: \"39539848-97ad-40d8-8082-d27a15143388\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p"
Oct 30 19:56:05.208072 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.201067    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/686b83f0-b8a4-4210-a66d-e92c30e7fc8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4\" (UID: \"686b83f0-b8a4-4210-a66d-e92c30e7fc8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4"
Oct 30 19:56:05.340114 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.335519    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/686b83f0-b8a4-4210-a66d-e92c30e7fc8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4\" (UID: \"686b83f0-b8a4-4210-a66d-e92c30e7fc8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4"
Oct 30 19:56:05.340114 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.335647    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twrxp\" (UniqueName: \"kubernetes.io/projected/39539848-97ad-40d8-8082-d27a15143388-kube-api-access-twrxp\") pod \"payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p\" (UID: \"39539848-97ad-40d8-8082-d27a15143388\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p"
Oct 30 19:56:05.340114 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.335806    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjns7\" (UniqueName: \"kubernetes.io/projected/686b83f0-b8a4-4210-a66d-e92c30e7fc8c-kube-api-access-mjns7\") pod \"payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4\" (UID: \"686b83f0-b8a4-4210-a66d-e92c30e7fc8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4"
Oct 30 19:56:05.340114 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.335846    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39539848-97ad-40d8-8082-d27a15143388-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p\" (UID: \"39539848-97ad-40d8-8082-d27a15143388\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p"
Oct 30 19:56:05.340114 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.336990    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39539848-97ad-40d8-8082-d27a15143388-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p\" (UID: \"39539848-97ad-40d8-8082-d27a15143388\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p"
Oct 30 19:56:05.340114 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.337777    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/686b83f0-b8a4-4210-a66d-e92c30e7fc8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4\" (UID: \"686b83f0-b8a4-4210-a66d-e92c30e7fc8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4"
Oct 30 19:56:05.382064 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.380103    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjns7\" (UniqueName: \"kubernetes.io/projected/686b83f0-b8a4-4210-a66d-e92c30e7fc8c-kube-api-access-mjns7\") pod \"payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4\" (UID: \"686b83f0-b8a4-4210-a66d-e92c30e7fc8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4"
Oct 30 19:56:05.405995 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.394673    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twrxp\" (UniqueName: \"kubernetes.io/projected/39539848-97ad-40d8-8082-d27a15143388-kube-api-access-twrxp\") pod \"payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p\" (UID: \"39539848-97ad-40d8-8082-d27a15143388\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p"
Oct 30 19:56:05.415300 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.409366    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4"
Oct 30 19:56:05.642179 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.642090    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p"
Oct 30 19:56:05.811055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.811016    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4"]
Oct 30 19:56:05.811805 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:56:05.811784    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686b83f0_b8a4_4210_a66d_e92c30e7fc8c.slice/crio-5d45f524d27412c86c57b73403e6081f0d954478beacab7899bb2adbc80a2a8a WatchSource:0}: Error finding container 5d45f524d27412c86c57b73403e6081f0d954478beacab7899bb2adbc80a2a8a: Status 404 returned error can't find the container with id 5d45f524d27412c86c57b73403e6081f0d954478beacab7899bb2adbc80a2a8a
Oct 30 19:56:05.944837 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:05.944809    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p"]
Oct 30 19:56:05.945388 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:56:05.945360    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39539848_97ad_40d8_8082_d27a15143388.slice/crio-ebbb8460ee9e3510ac9f176bba1d990e0f5598bfd71aba1b96a5dc06a34119e1 WatchSource:0}: Error finding container ebbb8460ee9e3510ac9f176bba1d990e0f5598bfd71aba1b96a5dc06a34119e1: Status 404 returned error can't find the container with id ebbb8460ee9e3510ac9f176bba1d990e0f5598bfd71aba1b96a5dc06a34119e1
Oct 30 19:56:06.752810 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:06.752765    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p" event={"ID":"39539848-97ad-40d8-8082-d27a15143388","Type":"ContainerStarted","Data":"029abd593bf48e9381e4f0ca4707219c3638238d28dd49f2de6ac2482231c1d1"}
Oct 30 19:56:06.753001 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:06.752835    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p" event={"ID":"39539848-97ad-40d8-8082-d27a15143388","Type":"ContainerStarted","Data":"ebbb8460ee9e3510ac9f176bba1d990e0f5598bfd71aba1b96a5dc06a34119e1"}
Oct 30 19:56:06.753107 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:06.753000    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p"
Oct 30 19:56:06.754566 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:06.754536    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4" event={"ID":"686b83f0-b8a4-4210-a66d-e92c30e7fc8c","Type":"ContainerStarted","Data":"15b2c4ed74764dc5419c6a673ef511d8b24cbe5b8de5a81b4ffdeed3511b56e2"}
Oct 30 19:56:06.754681 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:06.754588    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4" event={"ID":"686b83f0-b8a4-4210-a66d-e92c30e7fc8c","Type":"ContainerStarted","Data":"5d45f524d27412c86c57b73403e6081f0d954478beacab7899bb2adbc80a2a8a"}
Oct 30 19:56:06.754681 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:06.754654    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4"
Oct 30 19:56:06.774803 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:06.774756    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p" podStartSLOduration=2.774742619 podStartE2EDuration="2.774742619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:06.772500066 +0000 UTC m=+1521.356769683" watchObservedRunningTime="2026-10-30 19:56:06.774742619 +0000 UTC m=+1521.359012237"
Oct 30 19:56:06.800457 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:06.800410    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4" podStartSLOduration=1.800394196 podStartE2EDuration="1.800394196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:06.799601123 +0000 UTC m=+1521.383870739" watchObservedRunningTime="2026-10-30 19:56:06.800394196 +0000 UTC m=+1521.384663813"
Oct 30 19:56:09.774756 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:56:09.773728    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:56:09.945984 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:09.945953    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"]
Oct 30 19:56:09.946271 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:09.946240    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69" podUID="42281754-675b-4589-8ce6-80ec15bda7a3" containerName="maas-api" containerID="cri-o://92f9c3280a4a6269420abd8058dce18bf3d73c63010e782830df396763272968" gracePeriod=30
Oct 30 19:56:10.660064 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:10.659223    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"
Oct 30 19:56:10.810820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:10.805840    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42281754-675b-4589-8ce6-80ec15bda7a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42281754-675b-4589-8ce6-80ec15bda7a3-maas-api-tls\") pod \"42281754-675b-4589-8ce6-80ec15bda7a3\" (UID: \"42281754-675b-4589-8ce6-80ec15bda7a3\") "
Oct 30 19:56:10.810820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:10.806052    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42281754-675b-4589-8ce6-80ec15bda7a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/42281754-675b-4589-8ce6-80ec15bda7a3-metrics-certs\") pod \"42281754-675b-4589-8ce6-80ec15bda7a3\" (UID: \"42281754-675b-4589-8ce6-80ec15bda7a3\") "
Oct 30 19:56:10.810820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:10.806172    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42281754-675b-4589-8ce6-80ec15bda7a3-kube-api-access-fh586\" (UniqueName: \"kubernetes.io/projected/42281754-675b-4589-8ce6-80ec15bda7a3-kube-api-access-fh586\") pod \"42281754-675b-4589-8ce6-80ec15bda7a3\" (UID: \"42281754-675b-4589-8ce6-80ec15bda7a3\") "
Oct 30 19:56:10.810820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:10.806339    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42281754-675b-4589-8ce6-80ec15bda7a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42281754-675b-4589-8ce6-80ec15bda7a3-openshift-service-ca\") pod \"42281754-675b-4589-8ce6-80ec15bda7a3\" (UID: \"42281754-675b-4589-8ce6-80ec15bda7a3\") "
Oct 30 19:56:10.810820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:10.808975    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42281754-675b-4589-8ce6-80ec15bda7a3-openshift-service-ca" pod "42281754-675b-4589-8ce6-80ec15bda7a3" (UID: "42281754-675b-4589-8ce6-80ec15bda7a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:10.826108 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:10.815517    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42281754-675b-4589-8ce6-80ec15bda7a3-kube-api-access-fh586" pod "42281754-675b-4589-8ce6-80ec15bda7a3" (UID: "42281754-675b-4589-8ce6-80ec15bda7a3"). InnerVolumeSpecName "kube-api-access-fh586". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:10.826495 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:10.826465    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"
Oct 30 19:56:10.848787 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:10.848738    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42281754-675b-4589-8ce6-80ec15bda7a3-maas-api-tls" pod "42281754-675b-4589-8ce6-80ec15bda7a3" (UID: "42281754-675b-4589-8ce6-80ec15bda7a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:11.050683 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:11.043150    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42281754-675b-4589-8ce6-80ec15bda7a3-metrics-certs" pod "42281754-675b-4589-8ce6-80ec15bda7a3" (UID: "42281754-675b-4589-8ce6-80ec15bda7a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:11.050683 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:11.048636    5408 generic.go:356] "Generic (PLEG): container finished" podID="42281754-675b-4589-8ce6-80ec15bda7a3" containerID="92f9c3280a4a6269420abd8058dce18bf3d73c63010e782830df396763272968" exitCode=0
Oct 30 19:56:11.050683 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:11.048743    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69" event={"ID":"42281754-675b-4589-8ce6-80ec15bda7a3","Type":"ContainerDied","Data":"92f9c3280a4a6269420abd8058dce18bf3d73c63010e782830df396763272968"}
Oct 30 19:56:11.050683 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:11.048795    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69" event={"ID":"42281754-675b-4589-8ce6-80ec15bda7a3","Type":"ContainerDied","Data":"1e6779117c5f0d23f6b64cd7188eb797bad0d8bf43e5acb58564007e4cefcb3e"}
Oct 30 19:56:11.050683 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:11.048822    5408 scope.go:122] "RemoveContainer" containerID="92f9c3280a4a6269420abd8058dce18bf3d73c63010e782830df396763272968"
Oct 30 19:56:11.050683 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:11.048988    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"
Oct 30 19:56:11.079043 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:11.071941    5408 scope.go:122] "RemoveContainer" containerID="92f9c3280a4a6269420abd8058dce18bf3d73c63010e782830df396763272968"
Oct 30 19:56:11.079043 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:56:11.072368    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f9c3280a4a6269420abd8058dce18bf3d73c63010e782830df396763272968\": container with ID starting with 92f9c3280a4a6269420abd8058dce18bf3d73c63010e782830df396763272968 not found: ID does not exist" containerID="92f9c3280a4a6269420abd8058dce18bf3d73c63010e782830df396763272968"
Oct 30 19:56:11.079043 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:11.072402    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f9c3280a4a6269420abd8058dce18bf3d73c63010e782830df396763272968"} err="failed to get container status \"92f9c3280a4a6269420abd8058dce18bf3d73c63010e782830df396763272968\": rpc error: code = NotFound desc = could not find container \"92f9c3280a4a6269420abd8058dce18bf3d73c63010e782830df396763272968\": container with ID starting with 92f9c3280a4a6269420abd8058dce18bf3d73c63010e782830df396763272968 not found: ID does not exist"
Oct 30 19:56:11.094706 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:11.094671    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh586\" (UniqueName: \"kubernetes.io/projected/42281754-675b-4589-8ce6-80ec15bda7a3-kube-api-access-fh586\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:11.176729 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:11.176699    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42281754-675b-4589-8ce6-80ec15bda7a3-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:11.176947 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:11.176934    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42281754-675b-4589-8ce6-80ec15bda7a3-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:11.177056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:11.177043    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42281754-675b-4589-8ce6-80ec15bda7a3-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:11.390112 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:11.379371    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"]
Oct 30 19:56:11.448065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:11.446479    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d07cb404-744cc69d4d-6bg69"]
Oct 30 19:56:11.538335 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:11.538296    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42281754-675b-4589-8ce6-80ec15bda7a3" path="/var/lib/kubelet/pods/42281754-675b-4589-8ce6-80ec15bda7a3/volumes"
Oct 30 19:56:15.152190 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.152153    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568"]
Oct 30 19:56:15.153276 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.153255    5408 cpu_manager.go:414] "containerMap: removing container" podUID="42281754-675b-4589-8ce6-80ec15bda7a3" containerName="maas-api"
Oct 30 19:56:15.153341 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.153279    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42281754-675b-4589-8ce6-80ec15bda7a3" containerName="maas-api"
Oct 30 19:56:15.153458 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.153444    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="42281754-675b-4589-8ce6-80ec15bda7a3" containerName="maas-api"
Oct 30 19:56:15.158849 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.158828    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568"
Oct 30 19:56:15.171850 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.171822    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568"]
Oct 30 19:56:15.184944 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.184909    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e7f2a520\""
Oct 30 19:56:15.189361 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.189334    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh75j\" (UniqueName: \"kubernetes.io/projected/733ad0ea-4b8f-4739-b7b8-bc32f148028b-kube-api-access-qh75j\") pod \"payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568\" (UID: \"733ad0ea-4b8f-4739-b7b8-bc32f148028b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568"
Oct 30 19:56:15.189469 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.189388    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733ad0ea-4b8f-4739-b7b8-bc32f148028b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568\" (UID: \"733ad0ea-4b8f-4739-b7b8-bc32f148028b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568"
Oct 30 19:56:15.234997 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.234974    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf"]
Oct 30 19:56:15.239014 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.238998    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf"
Oct 30 19:56:15.273714 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.273683    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf"]
Oct 30 19:56:15.309388 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.309356    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e7f2a520-dockercfg-grgpl\""
Oct 30 19:56:15.379053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.311052    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh75j\" (UniqueName: \"kubernetes.io/projected/733ad0ea-4b8f-4739-b7b8-bc32f148028b-kube-api-access-qh75j\") pod \"payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568\" (UID: \"733ad0ea-4b8f-4739-b7b8-bc32f148028b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568"
Oct 30 19:56:15.379053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.363324    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733ad0ea-4b8f-4739-b7b8-bc32f148028b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568\" (UID: \"733ad0ea-4b8f-4739-b7b8-bc32f148028b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568"
Oct 30 19:56:15.379053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.364371    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733ad0ea-4b8f-4739-b7b8-bc32f148028b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568\" (UID: \"733ad0ea-4b8f-4739-b7b8-bc32f148028b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568"
Oct 30 19:56:15.480754 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.480666    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nzzw\" (UniqueName: \"kubernetes.io/projected/7e7d4ffa-a1ee-416c-b9cf-428f3762c81c-kube-api-access-4nzzw\") pod \"payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf\" (UID: \"7e7d4ffa-a1ee-416c-b9cf-428f3762c81c\") " pod="openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf"
Oct 30 19:56:15.481221 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.481198    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e7d4ffa-a1ee-416c-b9cf-428f3762c81c-plugins-config-volume\") pod \"payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf\" (UID: \"7e7d4ffa-a1ee-416c-b9cf-428f3762c81c\") " pod="openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf"
Oct 30 19:56:15.502057 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.488724    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh75j\" (UniqueName: \"kubernetes.io/projected/733ad0ea-4b8f-4739-b7b8-bc32f148028b-kube-api-access-qh75j\") pod \"payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568\" (UID: \"733ad0ea-4b8f-4739-b7b8-bc32f148028b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568"
Oct 30 19:56:15.587047 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.586985    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nzzw\" (UniqueName: \"kubernetes.io/projected/7e7d4ffa-a1ee-416c-b9cf-428f3762c81c-kube-api-access-4nzzw\") pod \"payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf\" (UID: \"7e7d4ffa-a1ee-416c-b9cf-428f3762c81c\") " pod="openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf"
Oct 30 19:56:15.587360 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.587334    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e7d4ffa-a1ee-416c-b9cf-428f3762c81c-plugins-config-volume\") pod \"payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf\" (UID: \"7e7d4ffa-a1ee-416c-b9cf-428f3762c81c\") " pod="openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf"
Oct 30 19:56:15.588793 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.588765    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e7d4ffa-a1ee-416c-b9cf-428f3762c81c-plugins-config-volume\") pod \"payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf\" (UID: \"7e7d4ffa-a1ee-416c-b9cf-428f3762c81c\") " pod="openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf"
Oct 30 19:56:15.658549 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.658097    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nzzw\" (UniqueName: \"kubernetes.io/projected/7e7d4ffa-a1ee-416c-b9cf-428f3762c81c-kube-api-access-4nzzw\") pod \"payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf\" (UID: \"7e7d4ffa-a1ee-416c-b9cf-428f3762c81c\") " pod="openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf"
Oct 30 19:56:15.779826 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.779530    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568"
Oct 30 19:56:15.852238 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:15.852209    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf"
Oct 30 19:56:16.092368 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:16.092338    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568"]
Oct 30 19:56:16.094102 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:56:16.094080    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733ad0ea_4b8f_4739_b7b8_bc32f148028b.slice/crio-aa559d2dfceeada1c4fbe737c2684e45c1d006372618eab16d5c87bd9ffc117d WatchSource:0}: Error finding container aa559d2dfceeada1c4fbe737c2684e45c1d006372618eab16d5c87bd9ffc117d: Status 404 returned error can't find the container with id aa559d2dfceeada1c4fbe737c2684e45c1d006372618eab16d5c87bd9ffc117d
Oct 30 19:56:16.170124 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:16.170100    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf"]
Oct 30 19:56:16.173841 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:56:16.173817    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7d4ffa_a1ee_416c_b9cf_428f3762c81c.slice/crio-1a5ba76917bcd5c818c256406111ea5815faa32be67f81fbc1c20441e4b03fe4 WatchSource:0}: Error finding container 1a5ba76917bcd5c818c256406111ea5815faa32be67f81fbc1c20441e4b03fe4: Status 404 returned error can't find the container with id 1a5ba76917bcd5c818c256406111ea5815faa32be67f81fbc1c20441e4b03fe4
Oct 30 19:56:17.129051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:17.119892    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf" event={"ID":"7e7d4ffa-a1ee-416c-b9cf-428f3762c81c","Type":"ContainerStarted","Data":"4d99922f98f30af6734e6d7dd40b02d26e9cbfe60ccc93b52568b2031c556fe3"}
Oct 30 19:56:17.129051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:17.119961    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf" event={"ID":"7e7d4ffa-a1ee-416c-b9cf-428f3762c81c","Type":"ContainerStarted","Data":"1a5ba76917bcd5c818c256406111ea5815faa32be67f81fbc1c20441e4b03fe4"}
Oct 30 19:56:17.129051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:17.121207    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf"
Oct 30 19:56:17.129051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:17.124162    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568" event={"ID":"733ad0ea-4b8f-4739-b7b8-bc32f148028b","Type":"ContainerStarted","Data":"3a150e51c9dfb8c606fd0edaa69fc89f76525fe9450082cd133cd0a53e49e3cc"}
Oct 30 19:56:17.129051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:17.124229    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568" event={"ID":"733ad0ea-4b8f-4739-b7b8-bc32f148028b","Type":"ContainerStarted","Data":"aa559d2dfceeada1c4fbe737c2684e45c1d006372618eab16d5c87bd9ffc117d"}
Oct 30 19:56:17.129051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:17.125174    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568"
Oct 30 19:56:17.185539 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:17.185489    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf" podStartSLOduration=2.185475061 podStartE2EDuration="2.185475061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:17.150611098 +0000 UTC m=+1531.734880715" watchObservedRunningTime="2026-10-30 19:56:17.185475061 +0000 UTC m=+1531.769744699"
Oct 30 19:56:17.187119 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:17.187087    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568" podStartSLOduration=2.187076587 podStartE2EDuration="2.187076587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:17.182948272 +0000 UTC m=+1531.767217890" watchObservedRunningTime="2026-10-30 19:56:17.187076587 +0000 UTC m=+1531.771346206"
Oct 30 19:56:17.871050 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:17.866236    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p"
Oct 30 19:56:17.882149 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:17.882019    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4"
Oct 30 19:56:29.167107 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:29.166262    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568"
Oct 30 19:56:29.188065 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:29.168276    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf"
Oct 30 19:56:31.588058 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:31.587226    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:31.588058 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:31.587631    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Oct 30 19:56:31.637051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:31.589445    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:31.637051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:31.609453    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:31.637051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:31.619095    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:31.803052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:31.799114    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh"]
Oct 30 19:56:31.839114 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:31.826046    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf"]
Oct 30 19:56:31.902057 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:31.891209    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:31.902057 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:31.899949    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:31.927060 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:31.926252    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:32.413053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:32.407224    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:32.609083 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:32.606178    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:32.610655 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:32.609900    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:56:32.925516 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:32.924850    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:56:32.950050 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:32.925897    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:56:32.950050 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:32.938475    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:56:32.950050 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:32.938542    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:56:32.952040 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:32.951444    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:32.982055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:32.952258    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:56:33.071164 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:33.071119    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:33.334959 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:33.334928    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 19:56:33.503770 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:33.503739    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 19:56:33.572108 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:33.518164    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:56:33.705265 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:33.705162    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:33.764119 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:33.760533    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:56:33.764438 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:33.764420    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 19:56:34.100053 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:34.081939    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 19:56:34.147064 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:34.139845    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 19:56:36.799511 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:36.799461    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh" podUID="06ea6a85-2031-4c7a-ad94-9055c6a8f10a" containerName="payload-processing" containerID="cri-o://822dfa1d5e5e2b79cc706f6b697a4691cb6a635137961c7e206a443960dca739" gracePeriod=25
Oct 30 19:56:36.826828 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:36.826766    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf" podUID="a7102374-9148-4b5d-b21a-6b5b00d5022a" containerName="payload-pre-processing" containerID="cri-o://19de52b437b211a660bfd1379014f900e7dcf727a8342207302bc379ed861d37" gracePeriod=25
Oct 30 19:56:37.049966 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:37.049895    5408 generic.go:356] "Generic (PLEG): container finished" podID="a7102374-9148-4b5d-b21a-6b5b00d5022a" containerID="19de52b437b211a660bfd1379014f900e7dcf727a8342207302bc379ed861d37" exitCode=0
Oct 30 19:56:37.050155 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:37.049970    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf" event={"ID":"a7102374-9148-4b5d-b21a-6b5b00d5022a","Type":"ContainerDied","Data":"19de52b437b211a660bfd1379014f900e7dcf727a8342207302bc379ed861d37"}
Oct 30 19:56:37.053144 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:37.052931    5408 generic.go:356] "Generic (PLEG): container finished" podID="06ea6a85-2031-4c7a-ad94-9055c6a8f10a" containerID="822dfa1d5e5e2b79cc706f6b697a4691cb6a635137961c7e206a443960dca739" exitCode=0
Oct 30 19:56:37.053144 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:37.052986    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh" event={"ID":"06ea6a85-2031-4c7a-ad94-9055c6a8f10a","Type":"ContainerDied","Data":"822dfa1d5e5e2b79cc706f6b697a4691cb6a635137961c7e206a443960dca739"}
Oct 30 19:56:37.179537 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:37.179513    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh"
Oct 30 19:56:37.183377 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:37.183349    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf"
Oct 30 19:56:37.238159 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:37.238106    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ea6a85-2031-4c7a-ad94-9055c6a8f10a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ea6a85-2031-4c7a-ad94-9055c6a8f10a-plugins-config-volume\") pod \"06ea6a85-2031-4c7a-ad94-9055c6a8f10a\" (UID: \"06ea6a85-2031-4c7a-ad94-9055c6a8f10a\") "
Oct 30 19:56:37.238377 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:37.238163    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ea6a85-2031-4c7a-ad94-9055c6a8f10a-kube-api-access-fpmtl\" (UniqueName: \"kubernetes.io/projected/06ea6a85-2031-4c7a-ad94-9055c6a8f10a-kube-api-access-fpmtl\") pod \"06ea6a85-2031-4c7a-ad94-9055c6a8f10a\" (UID: \"06ea6a85-2031-4c7a-ad94-9055c6a8f10a\") "
Oct 30 19:56:37.238377 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:37.238218    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7102374-9148-4b5d-b21a-6b5b00d5022a-kube-api-access-2lbc6\" (UniqueName: \"kubernetes.io/projected/a7102374-9148-4b5d-b21a-6b5b00d5022a-kube-api-access-2lbc6\") pod \"a7102374-9148-4b5d-b21a-6b5b00d5022a\" (UID: \"a7102374-9148-4b5d-b21a-6b5b00d5022a\") "
Oct 30 19:56:37.238377 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:37.238373    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7102374-9148-4b5d-b21a-6b5b00d5022a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7102374-9148-4b5d-b21a-6b5b00d5022a-plugins-config-volume\") pod \"a7102374-9148-4b5d-b21a-6b5b00d5022a\" (UID: \"a7102374-9148-4b5d-b21a-6b5b00d5022a\") "
Oct 30 19:56:37.241276 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:37.240541    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ea6a85-2031-4c7a-ad94-9055c6a8f10a-plugins-config-volume" pod "06ea6a85-2031-4c7a-ad94-9055c6a8f10a" (UID: "06ea6a85-2031-4c7a-ad94-9055c6a8f10a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:37.241276 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:37.240990    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7102374-9148-4b5d-b21a-6b5b00d5022a-plugins-config-volume" pod "a7102374-9148-4b5d-b21a-6b5b00d5022a" (UID: "a7102374-9148-4b5d-b21a-6b5b00d5022a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:37.242957 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:37.242935    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ea6a85-2031-4c7a-ad94-9055c6a8f10a-kube-api-access-fpmtl" pod "06ea6a85-2031-4c7a-ad94-9055c6a8f10a" (UID: "06ea6a85-2031-4c7a-ad94-9055c6a8f10a"). InnerVolumeSpecName "kube-api-access-fpmtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:37.252795 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:37.252768    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7102374-9148-4b5d-b21a-6b5b00d5022a-kube-api-access-2lbc6" pod "a7102374-9148-4b5d-b21a-6b5b00d5022a" (UID: "a7102374-9148-4b5d-b21a-6b5b00d5022a"). InnerVolumeSpecName "kube-api-access-2lbc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:37.352872 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:37.352648    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ea6a85-2031-4c7a-ad94-9055c6a8f10a-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:37.352872 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:37.352692    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpmtl\" (UniqueName: \"kubernetes.io/projected/06ea6a85-2031-4c7a-ad94-9055c6a8f10a-kube-api-access-fpmtl\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:37.352872 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:37.352707    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lbc6\" (UniqueName: \"kubernetes.io/projected/a7102374-9148-4b5d-b21a-6b5b00d5022a-kube-api-access-2lbc6\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:37.352872 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:37.352722    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7102374-9148-4b5d-b21a-6b5b00d5022a-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:38.060568 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:38.060541    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf"
Oct 30 19:56:38.061069 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:38.060563    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf" event={"ID":"a7102374-9148-4b5d-b21a-6b5b00d5022a","Type":"ContainerDied","Data":"b244ddb9a90e2abb4412f49ba1baa43bc9389b3045aa4601556f7869d72e7cd5"}
Oct 30 19:56:38.061069 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:38.060633    5408 scope.go:122] "RemoveContainer" containerID="19de52b437b211a660bfd1379014f900e7dcf727a8342207302bc379ed861d37"
Oct 30 19:56:38.062070 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:38.062002    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh"
Oct 30 19:56:38.062070 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:38.062019    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh" event={"ID":"06ea6a85-2031-4c7a-ad94-9055c6a8f10a","Type":"ContainerDied","Data":"4248a4194fe09c81d43db656dd1d2e80562b4df34990fa07b48796f6c9b9b26f"}
Oct 30 19:56:38.070902 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:38.070886    5408 scope.go:122] "RemoveContainer" containerID="822dfa1d5e5e2b79cc706f6b697a4691cb6a635137961c7e206a443960dca739"
Oct 30 19:56:38.089260 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:38.089233    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh"]
Oct 30 19:56:38.098263 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:38.098242    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b69bf89d-6f4d64b6bd-86jqh"]
Oct 30 19:56:38.114280 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:38.114259    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf"]
Oct 30 19:56:38.122475 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:38.122450    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b69bf89d-7bf65957c5-2jvjf"]
Oct 30 19:56:39.537400 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:39.537366    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ea6a85-2031-4c7a-ad94-9055c6a8f10a" path="/var/lib/kubelet/pods/06ea6a85-2031-4c7a-ad94-9055c6a8f10a/volumes"
Oct 30 19:56:39.538911 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:39.538887    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7102374-9148-4b5d-b21a-6b5b00d5022a" path="/var/lib/kubelet/pods/a7102374-9148-4b5d-b21a-6b5b00d5022a/volumes"
Oct 30 19:56:39.859785 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:56:39.859696    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:56:42.498703 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.498675    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v"]
Oct 30 19:56:42.499342 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.499325    5408 cpu_manager.go:414] "containerMap: removing container" podUID="06ea6a85-2031-4c7a-ad94-9055c6a8f10a" containerName="payload-processing"
Oct 30 19:56:42.499397 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.499343    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ea6a85-2031-4c7a-ad94-9055c6a8f10a" containerName="payload-processing"
Oct 30 19:56:42.499397 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.499356    5408 cpu_manager.go:414] "containerMap: removing container" podUID="a7102374-9148-4b5d-b21a-6b5b00d5022a" containerName="payload-pre-processing"
Oct 30 19:56:42.499397 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.499361    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7102374-9148-4b5d-b21a-6b5b00d5022a" containerName="payload-pre-processing"
Oct 30 19:56:42.499500 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.499493    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ea6a85-2031-4c7a-ad94-9055c6a8f10a" containerName="payload-processing"
Oct 30 19:56:42.499542 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.499503    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7102374-9148-4b5d-b21a-6b5b00d5022a" containerName="payload-pre-processing"
Oct 30 19:56:42.504577 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.504557    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v"
Oct 30 19:56:42.516440 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.516421    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-f5d176\""
Oct 30 19:56:42.524038 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.524009    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v"]
Oct 30 19:56:42.524473 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.524453    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-f5d176-dockercfg-2k7jx\""
Oct 30 19:56:42.563386 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.563350    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll"]
Oct 30 19:56:42.568947 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.568921    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll"
Oct 30 19:56:42.586501 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.586477    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0bb64d8-6abf-45cc-8feb-d7ba98073055-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v\" (UID: \"c0bb64d8-6abf-45cc-8feb-d7ba98073055\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v"
Oct 30 19:56:42.586630 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.586549    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fsz4\" (UniqueName: \"kubernetes.io/projected/c0bb64d8-6abf-45cc-8feb-d7ba98073055-kube-api-access-5fsz4\") pod \"payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v\" (UID: \"c0bb64d8-6abf-45cc-8feb-d7ba98073055\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v"
Oct 30 19:56:42.590002 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.589534    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll"]
Oct 30 19:56:42.705051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.703321    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0bb64d8-6abf-45cc-8feb-d7ba98073055-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v\" (UID: \"c0bb64d8-6abf-45cc-8feb-d7ba98073055\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v"
Oct 30 19:56:42.705051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.703431    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fsz4\" (UniqueName: \"kubernetes.io/projected/c0bb64d8-6abf-45cc-8feb-d7ba98073055-kube-api-access-5fsz4\") pod \"payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v\" (UID: \"c0bb64d8-6abf-45cc-8feb-d7ba98073055\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v"
Oct 30 19:56:42.705051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.703489    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvn9t\" (UniqueName: \"kubernetes.io/projected/a848a44a-9e0a-47aa-811f-5a9930b9bd9f-kube-api-access-dvn9t\") pod \"payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll\" (UID: \"a848a44a-9e0a-47aa-811f-5a9930b9bd9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll"
Oct 30 19:56:42.705051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.703543    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a848a44a-9e0a-47aa-811f-5a9930b9bd9f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll\" (UID: \"a848a44a-9e0a-47aa-811f-5a9930b9bd9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll"
Oct 30 19:56:42.732149 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.732117    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0bb64d8-6abf-45cc-8feb-d7ba98073055-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v\" (UID: \"c0bb64d8-6abf-45cc-8feb-d7ba98073055\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v"
Oct 30 19:56:42.745054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.736005    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fsz4\" (UniqueName: \"kubernetes.io/projected/c0bb64d8-6abf-45cc-8feb-d7ba98073055-kube-api-access-5fsz4\") pod \"payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v\" (UID: \"c0bb64d8-6abf-45cc-8feb-d7ba98073055\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v"
Oct 30 19:56:42.806107 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.805273    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvn9t\" (UniqueName: \"kubernetes.io/projected/a848a44a-9e0a-47aa-811f-5a9930b9bd9f-kube-api-access-dvn9t\") pod \"payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll\" (UID: \"a848a44a-9e0a-47aa-811f-5a9930b9bd9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll"
Oct 30 19:56:42.806107 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.805356    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a848a44a-9e0a-47aa-811f-5a9930b9bd9f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll\" (UID: \"a848a44a-9e0a-47aa-811f-5a9930b9bd9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll"
Oct 30 19:56:42.818052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.806677    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a848a44a-9e0a-47aa-811f-5a9930b9bd9f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll\" (UID: \"a848a44a-9e0a-47aa-811f-5a9930b9bd9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll"
Oct 30 19:56:42.822862 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.819494    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v"
Oct 30 19:56:42.873954 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.873915    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvn9t\" (UniqueName: \"kubernetes.io/projected/a848a44a-9e0a-47aa-811f-5a9930b9bd9f-kube-api-access-dvn9t\") pod \"payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll\" (UID: \"a848a44a-9e0a-47aa-811f-5a9930b9bd9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll"
Oct 30 19:56:42.893624 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:42.893373    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll"
Oct 30 19:56:43.309960 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:43.309718    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll"]
Oct 30 19:56:43.312651 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:56:43.312626    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda848a44a_9e0a_47aa_811f_5a9930b9bd9f.slice/crio-7a9d87d1cdb6b555c9801d283545d5b52371fe55024b27c2991bdd36635e3f03 WatchSource:0}: Error finding container 7a9d87d1cdb6b555c9801d283545d5b52371fe55024b27c2991bdd36635e3f03: Status 404 returned error can't find the container with id 7a9d87d1cdb6b555c9801d283545d5b52371fe55024b27c2991bdd36635e3f03
Oct 30 19:56:43.314931 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:43.314909    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v"]
Oct 30 19:56:43.315063 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:56:43.315009    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0bb64d8_6abf_45cc_8feb_d7ba98073055.slice/crio-4d65213267e6e089d6c14194dde014c2d4f9cd1a4ea37904c528f3a3c79ae446 WatchSource:0}: Error finding container 4d65213267e6e089d6c14194dde014c2d4f9cd1a4ea37904c528f3a3c79ae446: Status 404 returned error can't find the container with id 4d65213267e6e089d6c14194dde014c2d4f9cd1a4ea37904c528f3a3c79ae446
Oct 30 19:56:44.130230 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.130197    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll" event={"ID":"a848a44a-9e0a-47aa-811f-5a9930b9bd9f","Type":"ContainerStarted","Data":"98577692088c491a4e4b123552a7ba3b4c21731718062364c3caece3865a6b11"}
Oct 30 19:56:44.130658 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.130249    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll" event={"ID":"a848a44a-9e0a-47aa-811f-5a9930b9bd9f","Type":"ContainerStarted","Data":"7a9d87d1cdb6b555c9801d283545d5b52371fe55024b27c2991bdd36635e3f03"}
Oct 30 19:56:44.130658 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.130279    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll"
Oct 30 19:56:44.131783 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.131762    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v" event={"ID":"c0bb64d8-6abf-45cc-8feb-d7ba98073055","Type":"ContainerStarted","Data":"266adf1eda38260190b3dc1916ed5c639fbc6085d50fd1ddb19e5f8911187c71"}
Oct 30 19:56:44.131864 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.131799    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v" event={"ID":"c0bb64d8-6abf-45cc-8feb-d7ba98073055","Type":"ContainerStarted","Data":"4d65213267e6e089d6c14194dde014c2d4f9cd1a4ea37904c528f3a3c79ae446"}
Oct 30 19:56:44.131864 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.131820    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v"
Oct 30 19:56:44.150838 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.150797    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll" podStartSLOduration=2.150779678 podStartE2EDuration="2.150779678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:44.149079161 +0000 UTC m=+1558.733348775" watchObservedRunningTime="2026-10-30 19:56:44.150779678 +0000 UTC m=+1558.735049296"
Oct 30 19:56:44.177204 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.177165    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v" podStartSLOduration=2.177153324 podStartE2EDuration="2.177153324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:44.175415004 +0000 UTC m=+1558.759684624" watchObservedRunningTime="2026-10-30 19:56:44.177153324 +0000 UTC m=+1558.761422941"
Oct 30 19:56:44.238898 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.238867    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"]
Oct 30 19:56:44.239248 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.239216    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m" podUID="8ba7faec-f569-467c-935b-1d66fb69f44d" containerName="maas-api" containerID="cri-o://99b3aae0a5889c799bb4e6c40a0c31fdeea29f5030cf99b9e8c62dc4c56cb727" gracePeriod=30
Oct 30 19:56:44.795633 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.795605    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"
Oct 30 19:56:44.863008 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.862977    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba7faec-f569-467c-935b-1d66fb69f44d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba7faec-f569-467c-935b-1d66fb69f44d-openshift-service-ca\") pod \"8ba7faec-f569-467c-935b-1d66fb69f44d\" (UID: \"8ba7faec-f569-467c-935b-1d66fb69f44d\") "
Oct 30 19:56:44.863206 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.863191    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ba7faec-f569-467c-935b-1d66fb69f44d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba7faec-f569-467c-935b-1d66fb69f44d-metrics-certs\") pod \"8ba7faec-f569-467c-935b-1d66fb69f44d\" (UID: \"8ba7faec-f569-467c-935b-1d66fb69f44d\") "
Oct 30 19:56:44.863267 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.863256    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba7faec-f569-467c-935b-1d66fb69f44d-kube-api-access-xwrhw\" (UniqueName: \"kubernetes.io/projected/8ba7faec-f569-467c-935b-1d66fb69f44d-kube-api-access-xwrhw\") pod \"8ba7faec-f569-467c-935b-1d66fb69f44d\" (UID: \"8ba7faec-f569-467c-935b-1d66fb69f44d\") "
Oct 30 19:56:44.863311 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.863286    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ba7faec-f569-467c-935b-1d66fb69f44d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba7faec-f569-467c-935b-1d66fb69f44d-maas-api-tls\") pod \"8ba7faec-f569-467c-935b-1d66fb69f44d\" (UID: \"8ba7faec-f569-467c-935b-1d66fb69f44d\") "
Oct 30 19:56:44.864407 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.864374    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba7faec-f569-467c-935b-1d66fb69f44d-openshift-service-ca" pod "8ba7faec-f569-467c-935b-1d66fb69f44d" (UID: "8ba7faec-f569-467c-935b-1d66fb69f44d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:44.865924 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.865872    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba7faec-f569-467c-935b-1d66fb69f44d-maas-api-tls" pod "8ba7faec-f569-467c-935b-1d66fb69f44d" (UID: "8ba7faec-f569-467c-935b-1d66fb69f44d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:44.866388 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.866361    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba7faec-f569-467c-935b-1d66fb69f44d-metrics-certs" pod "8ba7faec-f569-467c-935b-1d66fb69f44d" (UID: "8ba7faec-f569-467c-935b-1d66fb69f44d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:44.866481 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.866453    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba7faec-f569-467c-935b-1d66fb69f44d-kube-api-access-xwrhw" pod "8ba7faec-f569-467c-935b-1d66fb69f44d" (UID: "8ba7faec-f569-467c-935b-1d66fb69f44d"). InnerVolumeSpecName "kube-api-access-xwrhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:44.965431 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.965341    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba7faec-f569-467c-935b-1d66fb69f44d-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:44.965431 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.965379    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwrhw\" (UniqueName: \"kubernetes.io/projected/8ba7faec-f569-467c-935b-1d66fb69f44d-kube-api-access-xwrhw\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:44.965431 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.965391    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba7faec-f569-467c-935b-1d66fb69f44d-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:44.965431 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:44.965399    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba7faec-f569-467c-935b-1d66fb69f44d-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:45.161645 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:45.161611    5408 generic.go:356] "Generic (PLEG): container finished" podID="8ba7faec-f569-467c-935b-1d66fb69f44d" containerID="99b3aae0a5889c799bb4e6c40a0c31fdeea29f5030cf99b9e8c62dc4c56cb727" exitCode=0
Oct 30 19:56:45.162072 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:45.161674    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"
Oct 30 19:56:45.162072 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:45.161687    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m" event={"ID":"8ba7faec-f569-467c-935b-1d66fb69f44d","Type":"ContainerDied","Data":"99b3aae0a5889c799bb4e6c40a0c31fdeea29f5030cf99b9e8c62dc4c56cb727"}
Oct 30 19:56:45.162072 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:45.161740    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m" event={"ID":"8ba7faec-f569-467c-935b-1d66fb69f44d","Type":"ContainerDied","Data":"fc5884b924d01cd56938db6dfde20120f26488226b535d325c1100a92769612f"}
Oct 30 19:56:45.162072 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:45.161768    5408 scope.go:122] "RemoveContainer" containerID="99b3aae0a5889c799bb4e6c40a0c31fdeea29f5030cf99b9e8c62dc4c56cb727"
Oct 30 19:56:45.174289 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:45.173901    5408 scope.go:122] "RemoveContainer" containerID="99b3aae0a5889c799bb4e6c40a0c31fdeea29f5030cf99b9e8c62dc4c56cb727"
Oct 30 19:56:45.174289 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:56:45.174264    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b3aae0a5889c799bb4e6c40a0c31fdeea29f5030cf99b9e8c62dc4c56cb727\": container with ID starting with 99b3aae0a5889c799bb4e6c40a0c31fdeea29f5030cf99b9e8c62dc4c56cb727 not found: ID does not exist" containerID="99b3aae0a5889c799bb4e6c40a0c31fdeea29f5030cf99b9e8c62dc4c56cb727"
Oct 30 19:56:45.174490 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:45.174296    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b3aae0a5889c799bb4e6c40a0c31fdeea29f5030cf99b9e8c62dc4c56cb727"} err="failed to get container status \"99b3aae0a5889c799bb4e6c40a0c31fdeea29f5030cf99b9e8c62dc4c56cb727\": rpc error: code = NotFound desc = could not find container \"99b3aae0a5889c799bb4e6c40a0c31fdeea29f5030cf99b9e8c62dc4c56cb727\": container with ID starting with 99b3aae0a5889c799bb4e6c40a0c31fdeea29f5030cf99b9e8c62dc4c56cb727 not found: ID does not exist"
Oct 30 19:56:45.203286 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:45.203260    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"]
Oct 30 19:56:45.208620 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:45.208595    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b69bf89d-684f6d696b-zqb4m"]
Oct 30 19:56:45.600182 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:45.600150    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba7faec-f569-467c-935b-1d66fb69f44d" path="/var/lib/kubelet/pods/8ba7faec-f569-467c-935b-1d66fb69f44d/volumes"
Oct 30 19:56:51.590988 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.590339    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"]
Oct 30 19:56:51.591496 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.591339    5408 cpu_manager.go:414] "containerMap: removing container" podUID="8ba7faec-f569-467c-935b-1d66fb69f44d" containerName="maas-api"
Oct 30 19:56:51.591496 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.591360    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba7faec-f569-467c-935b-1d66fb69f44d" containerName="maas-api"
Oct 30 19:56:51.591718 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.591638    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba7faec-f569-467c-935b-1d66fb69f44d" containerName="maas-api"
Oct 30 19:56:51.597150 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.597131    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"
Oct 30 19:56:51.612987 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.612960    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-f5d176-serving-cert\""
Oct 30 19:56:51.615441 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.615420    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"]
Oct 30 19:56:51.660711 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.660676    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr\" (UID: \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"
Oct 30 19:56:51.660878 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.660846    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr\" (UID: \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"
Oct 30 19:56:51.661020 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.661001    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kjts\" (UniqueName: \"kubernetes.io/projected/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-kube-api-access-8kjts\") pod \"maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr\" (UID: \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"
Oct 30 19:56:51.661206 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.661184    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr\" (UID: \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"
Oct 30 19:56:51.769062 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.765127    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr\" (UID: \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"
Oct 30 19:56:51.769062 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.765377    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr\" (UID: \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"
Oct 30 19:56:51.769062 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.765466    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr\" (UID: \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"
Oct 30 19:56:51.769062 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.765564    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kjts\" (UniqueName: \"kubernetes.io/projected/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-kube-api-access-8kjts\") pod \"maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr\" (UID: \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"
Oct 30 19:56:51.782048 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.779310    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr\" (UID: \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"
Oct 30 19:56:51.790858 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.788249    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kjts\" (UniqueName: \"kubernetes.io/projected/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-kube-api-access-8kjts\") pod \"maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr\" (UID: \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"
Oct 30 19:56:51.794670 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.793679    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr\" (UID: \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"
Oct 30 19:56:51.800992 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.800836    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr\" (UID: \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"
Oct 30 19:56:51.911207 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:51.911132    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"
Oct 30 19:56:52.195906 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:52.195880    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"]
Oct 30 19:56:52.199514 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:56:52.199482    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a32800_c4b4_46eb_abcd_fee2c0d88a29.slice/crio-d12a2f3fd4b503942a00873c7f28609561f76d6b0687efb62462a21429e7a102 WatchSource:0}: Error finding container d12a2f3fd4b503942a00873c7f28609561f76d6b0687efb62462a21429e7a102: Status 404 returned error can't find the container with id d12a2f3fd4b503942a00873c7f28609561f76d6b0687efb62462a21429e7a102
Oct 30 19:56:52.259056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:52.246592    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr" event={"ID":"d7a32800-c4b4-46eb-abcd-fee2c0d88a29","Type":"ContainerStarted","Data":"d12a2f3fd4b503942a00873c7f28609561f76d6b0687efb62462a21429e7a102"}
Oct 30 19:56:53.254987 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:53.254949    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr" event={"ID":"d7a32800-c4b4-46eb-abcd-fee2c0d88a29","Type":"ContainerStarted","Data":"cfed23b36a0f925da80129baa357ef29476423016334000737a89f7a4b9e413d"}
Oct 30 19:56:53.255411 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:53.255147    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"
Oct 30 19:56:53.278667 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:53.278624    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr" podStartSLOduration=1.793940772 podStartE2EDuration="2.278611213s" totalImagesPullingTime="484.670441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:53.275818887 +0000 UTC m=+1567.860088504" watchObservedRunningTime="2026-10-30 19:56:53.278611213 +0000 UTC m=+1567.862880870"
Oct 30 19:56:55.165294 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:55.165263    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v"
Oct 30 19:56:55.165779 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:55.165456    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll"
Oct 30 19:56:55.344486 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:55.344455    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf"]
Oct 30 19:56:55.359178 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:55.359149    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568"]
Oct 30 19:56:59.309054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:56:59.308291    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"
Oct 30 19:57:00.345120 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:00.345080    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf" podUID="7e7d4ffa-a1ee-416c-b9cf-428f3762c81c" containerName="payload-processing" containerID="cri-o://4d99922f98f30af6734e6d7dd40b02d26e9cbfe60ccc93b52568b2031c556fe3" gracePeriod=25
Oct 30 19:57:00.364064 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:00.363124    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568" podUID="733ad0ea-4b8f-4739-b7b8-bc32f148028b" containerName="payload-pre-processing" containerID="cri-o://3a150e51c9dfb8c606fd0edaa69fc89f76525fe9450082cd133cd0a53e49e3cc" gracePeriod=25
Oct 30 19:57:00.908054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:00.904268    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568"
Oct 30 19:57:00.950057 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:00.933712    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733ad0ea-4b8f-4739-b7b8-bc32f148028b-kube-api-access-qh75j\" (UniqueName: \"kubernetes.io/projected/733ad0ea-4b8f-4739-b7b8-bc32f148028b-kube-api-access-qh75j\") pod \"733ad0ea-4b8f-4739-b7b8-bc32f148028b\" (UID: \"733ad0ea-4b8f-4739-b7b8-bc32f148028b\") "
Oct 30 19:57:00.950057 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:00.933890    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/733ad0ea-4b8f-4739-b7b8-bc32f148028b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733ad0ea-4b8f-4739-b7b8-bc32f148028b-plugins-config-volume\") pod \"733ad0ea-4b8f-4739-b7b8-bc32f148028b\" (UID: \"733ad0ea-4b8f-4739-b7b8-bc32f148028b\") "
Oct 30 19:57:00.950057 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:00.936041    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733ad0ea-4b8f-4739-b7b8-bc32f148028b-plugins-config-volume" pod "733ad0ea-4b8f-4739-b7b8-bc32f148028b" (UID: "733ad0ea-4b8f-4739-b7b8-bc32f148028b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:00.951357 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:00.951290    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733ad0ea-4b8f-4739-b7b8-bc32f148028b-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:00.997126 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:00.995710    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733ad0ea-4b8f-4739-b7b8-bc32f148028b-kube-api-access-qh75j" pod "733ad0ea-4b8f-4739-b7b8-bc32f148028b" (UID: "733ad0ea-4b8f-4739-b7b8-bc32f148028b"). InnerVolumeSpecName "kube-api-access-qh75j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:01.066054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.053364    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh75j\" (UniqueName: \"kubernetes.io/projected/733ad0ea-4b8f-4739-b7b8-bc32f148028b-kube-api-access-qh75j\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:01.143056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.137440    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf"
Oct 30 19:57:01.167583 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.154613    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e7d4ffa-a1ee-416c-b9cf-428f3762c81c-kube-api-access-4nzzw\" (UniqueName: \"kubernetes.io/projected/7e7d4ffa-a1ee-416c-b9cf-428f3762c81c-kube-api-access-4nzzw\") pod \"7e7d4ffa-a1ee-416c-b9cf-428f3762c81c\" (UID: \"7e7d4ffa-a1ee-416c-b9cf-428f3762c81c\") "
Oct 30 19:57:01.167583 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.155083    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e7d4ffa-a1ee-416c-b9cf-428f3762c81c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e7d4ffa-a1ee-416c-b9cf-428f3762c81c-plugins-config-volume\") pod \"7e7d4ffa-a1ee-416c-b9cf-428f3762c81c\" (UID: \"7e7d4ffa-a1ee-416c-b9cf-428f3762c81c\") "
Oct 30 19:57:01.167583 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.156523    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e7d4ffa-a1ee-416c-b9cf-428f3762c81c-plugins-config-volume" pod "7e7d4ffa-a1ee-416c-b9cf-428f3762c81c" (UID: "7e7d4ffa-a1ee-416c-b9cf-428f3762c81c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:01.208110 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.207983    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7d4ffa-a1ee-416c-b9cf-428f3762c81c-kube-api-access-4nzzw" pod "7e7d4ffa-a1ee-416c-b9cf-428f3762c81c" (UID: "7e7d4ffa-a1ee-416c-b9cf-428f3762c81c"). InnerVolumeSpecName "kube-api-access-4nzzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:01.258341 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.258255    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e7d4ffa-a1ee-416c-b9cf-428f3762c81c-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:01.258341 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.258288    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nzzw\" (UniqueName: \"kubernetes.io/projected/7e7d4ffa-a1ee-416c-b9cf-428f3762c81c-kube-api-access-4nzzw\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:01.330382 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.330351    5408 generic.go:356] "Generic (PLEG): container finished" podID="7e7d4ffa-a1ee-416c-b9cf-428f3762c81c" containerID="4d99922f98f30af6734e6d7dd40b02d26e9cbfe60ccc93b52568b2031c556fe3" exitCode=0
Oct 30 19:57:01.330559 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.330429    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf" event={"ID":"7e7d4ffa-a1ee-416c-b9cf-428f3762c81c","Type":"ContainerDied","Data":"4d99922f98f30af6734e6d7dd40b02d26e9cbfe60ccc93b52568b2031c556fe3"}
Oct 30 19:57:01.330559 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.330480    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf" event={"ID":"7e7d4ffa-a1ee-416c-b9cf-428f3762c81c","Type":"ContainerDied","Data":"1a5ba76917bcd5c818c256406111ea5815faa32be67f81fbc1c20441e4b03fe4"}
Oct 30 19:57:01.330559 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.330510    5408 scope.go:122] "RemoveContainer" containerID="4d99922f98f30af6734e6d7dd40b02d26e9cbfe60ccc93b52568b2031c556fe3"
Oct 30 19:57:01.330737 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.330656    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf"
Oct 30 19:57:01.335928 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.335900    5408 generic.go:356] "Generic (PLEG): container finished" podID="733ad0ea-4b8f-4739-b7b8-bc32f148028b" containerID="3a150e51c9dfb8c606fd0edaa69fc89f76525fe9450082cd133cd0a53e49e3cc" exitCode=0
Oct 30 19:57:01.336179 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.336051    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568" event={"ID":"733ad0ea-4b8f-4739-b7b8-bc32f148028b","Type":"ContainerDied","Data":"3a150e51c9dfb8c606fd0edaa69fc89f76525fe9450082cd133cd0a53e49e3cc"}
Oct 30 19:57:01.336179 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.336096    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568" event={"ID":"733ad0ea-4b8f-4739-b7b8-bc32f148028b","Type":"ContainerDied","Data":"aa559d2dfceeada1c4fbe737c2684e45c1d006372618eab16d5c87bd9ffc117d"}
Oct 30 19:57:01.336179 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.336162    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568"
Oct 30 19:57:01.357960 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.357944    5408 scope.go:122] "RemoveContainer" containerID="4d99922f98f30af6734e6d7dd40b02d26e9cbfe60ccc93b52568b2031c556fe3"
Oct 30 19:57:01.358964 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:57:01.358941    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d99922f98f30af6734e6d7dd40b02d26e9cbfe60ccc93b52568b2031c556fe3\": container with ID starting with 4d99922f98f30af6734e6d7dd40b02d26e9cbfe60ccc93b52568b2031c556fe3 not found: ID does not exist" containerID="4d99922f98f30af6734e6d7dd40b02d26e9cbfe60ccc93b52568b2031c556fe3"
Oct 30 19:57:01.359108 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.359087    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d99922f98f30af6734e6d7dd40b02d26e9cbfe60ccc93b52568b2031c556fe3"} err="failed to get container status \"4d99922f98f30af6734e6d7dd40b02d26e9cbfe60ccc93b52568b2031c556fe3\": rpc error: code = NotFound desc = could not find container \"4d99922f98f30af6734e6d7dd40b02d26e9cbfe60ccc93b52568b2031c556fe3\": container with ID starting with 4d99922f98f30af6734e6d7dd40b02d26e9cbfe60ccc93b52568b2031c556fe3 not found: ID does not exist"
Oct 30 19:57:01.359200 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.359189    5408 scope.go:122] "RemoveContainer" containerID="3a150e51c9dfb8c606fd0edaa69fc89f76525fe9450082cd133cd0a53e49e3cc"
Oct 30 19:57:01.378112 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.378088    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf"]
Oct 30 19:57:01.389332 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.386293    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e7f2a520-6d7b6c4c94-xhtgf"]
Oct 30 19:57:01.390611 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.390578    5408 scope.go:122] "RemoveContainer" containerID="3a150e51c9dfb8c606fd0edaa69fc89f76525fe9450082cd133cd0a53e49e3cc"
Oct 30 19:57:01.391741 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:57:01.391382    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a150e51c9dfb8c606fd0edaa69fc89f76525fe9450082cd133cd0a53e49e3cc\": container with ID starting with 3a150e51c9dfb8c606fd0edaa69fc89f76525fe9450082cd133cd0a53e49e3cc not found: ID does not exist" containerID="3a150e51c9dfb8c606fd0edaa69fc89f76525fe9450082cd133cd0a53e49e3cc"
Oct 30 19:57:01.391741 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.391407    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a150e51c9dfb8c606fd0edaa69fc89f76525fe9450082cd133cd0a53e49e3cc"} err="failed to get container status \"3a150e51c9dfb8c606fd0edaa69fc89f76525fe9450082cd133cd0a53e49e3cc\": rpc error: code = NotFound desc = could not find container \"3a150e51c9dfb8c606fd0edaa69fc89f76525fe9450082cd133cd0a53e49e3cc\": container with ID starting with 3a150e51c9dfb8c606fd0edaa69fc89f76525fe9450082cd133cd0a53e49e3cc not found: ID does not exist"
Oct 30 19:57:01.415557 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.415527    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568"]
Oct 30 19:57:01.415738 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.415577    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7f2a520-5b7dcfd68c-fr568"]
Oct 30 19:57:01.754886 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.754850    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733ad0ea-4b8f-4739-b7b8-bc32f148028b" path="/var/lib/kubelet/pods/733ad0ea-4b8f-4739-b7b8-bc32f148028b/volumes"
Oct 30 19:57:01.762009 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:01.761980    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e7d4ffa-a1ee-416c-b9cf-428f3762c81c" path="/var/lib/kubelet/pods/7e7d4ffa-a1ee-416c-b9cf-428f3762c81c/volumes"
Oct 30 19:57:04.604387 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:04.598127    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-h68ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:57:04.604387 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:04.598212    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls" podUID="e6ba43a0-ddfc-459d-8e82-1c2e197557a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:07.561657 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:07.561626    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11080-1f54c8"]
Oct 30 19:57:07.562296 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:07.562279    5408 cpu_manager.go:414] "containerMap: removing container" podUID="7e7d4ffa-a1ee-416c-b9cf-428f3762c81c" containerName="payload-processing"
Oct 30 19:57:07.562351 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:07.562296    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e7d4ffa-a1ee-416c-b9cf-428f3762c81c" containerName="payload-processing"
Oct 30 19:57:07.562351 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:07.562318    5408 cpu_manager.go:414] "containerMap: removing container" podUID="733ad0ea-4b8f-4739-b7b8-bc32f148028b" containerName="payload-pre-processing"
Oct 30 19:57:07.562351 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:07.562323    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733ad0ea-4b8f-4739-b7b8-bc32f148028b" containerName="payload-pre-processing"
Oct 30 19:57:07.562450 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:07.562402    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="733ad0ea-4b8f-4739-b7b8-bc32f148028b" containerName="payload-pre-processing"
Oct 30 19:57:07.562450 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:07.562414    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e7d4ffa-a1ee-416c-b9cf-428f3762c81c" containerName="payload-processing"
Oct 30 19:57:07.565944 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:07.565926    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-1f54c8"
Oct 30 19:57:07.581348 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:07.581322    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xqd5g\""
Oct 30 19:57:07.582566 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:07.582543    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11080-1f54c8"]
Oct 30 19:57:07.698493 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:07.698455    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bmhq\" (UniqueName: \"kubernetes.io/projected/abc54d37-ba6e-4915-a2d9-83970ec6ae30-kube-api-access-8bmhq\") pod \"test-curl-11080-1f54c8\" (UID: \"abc54d37-ba6e-4915-a2d9-83970ec6ae30\") " pod="opendatahub/test-curl-11080-1f54c8"
Oct 30 19:57:07.800304 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:07.800270    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bmhq\" (UniqueName: \"kubernetes.io/projected/abc54d37-ba6e-4915-a2d9-83970ec6ae30-kube-api-access-8bmhq\") pod \"test-curl-11080-1f54c8\" (UID: \"abc54d37-ba6e-4915-a2d9-83970ec6ae30\") " pod="opendatahub/test-curl-11080-1f54c8"
Oct 30 19:57:07.827289 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:07.827222    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bmhq\" (UniqueName: \"kubernetes.io/projected/abc54d37-ba6e-4915-a2d9-83970ec6ae30-kube-api-access-8bmhq\") pod \"test-curl-11080-1f54c8\" (UID: \"abc54d37-ba6e-4915-a2d9-83970ec6ae30\") " pod="opendatahub/test-curl-11080-1f54c8"
Oct 30 19:57:07.878425 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:07.878393    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-1f54c8"
Oct 30 19:57:08.177015 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:08.176988    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11080-1f54c8"]
Oct 30 19:57:08.177991 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:57:08.177935    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabc54d37_ba6e_4915_a2d9_83970ec6ae30.slice/crio-991afa34481b10d42d944abfd4bda21e9f22c7745fadeec0b8bfcb87302b68d4 WatchSource:0}: Error finding container 991afa34481b10d42d944abfd4bda21e9f22c7745fadeec0b8bfcb87302b68d4: Status 404 returned error can't find the container with id 991afa34481b10d42d944abfd4bda21e9f22c7745fadeec0b8bfcb87302b68d4
Oct 30 19:57:08.749933 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:08.749899    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11080-1f54c8" event={"ID":"abc54d37-ba6e-4915-a2d9-83970ec6ae30","Type":"ContainerStarted","Data":"393226e8cbf8d9c4768126719d6bdf3e7f01a6e0a6d29d6e0372415d141a9225"}
Oct 30 19:57:08.750351 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:08.749943    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11080-1f54c8" event={"ID":"abc54d37-ba6e-4915-a2d9-83970ec6ae30","Type":"ContainerStarted","Data":"991afa34481b10d42d944abfd4bda21e9f22c7745fadeec0b8bfcb87302b68d4"}
Oct 30 19:57:08.770728 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:08.770679    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11080-1f54c8" podStartSLOduration=1.770665897 podStartE2EDuration="1.770665897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:08.768153115 +0000 UTC m=+1583.352422733" watchObservedRunningTime="2026-10-30 19:57:08.770665897 +0000 UTC m=+1583.354935513"
Oct 30 19:57:09.599214 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:09.599166    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11080-1f54c8"]
Oct 30 19:57:09.613081 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:09.613048    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11080-1f54c8"]
Oct 30 19:57:09.666452 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:57:09.666410    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:57:09.826147 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:09.821476    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11080-33742e"]
Oct 30 19:57:09.827705 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:09.827678    5408 cpu_manager.go:414] "containerMap: removing container" podUID="abc54d37-ba6e-4915-a2d9-83970ec6ae30" containerName="test-curl-11080-1f54c8"
Oct 30 19:57:09.861368 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:09.861279    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc54d37-ba6e-4915-a2d9-83970ec6ae30" containerName="test-curl-11080-1f54c8"
Oct 30 19:57:09.861901 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:09.861878    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc54d37-ba6e-4915-a2d9-83970ec6ae30" containerName="test-curl-11080-1f54c8"
Oct 30 19:57:09.866898 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:09.866876    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11080-33742e"]
Oct 30 19:57:09.866989 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:09.866979    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-33742e"
Oct 30 19:57:09.963005 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:09.962970    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gtx5\" (UniqueName: \"kubernetes.io/projected/a33c8ea3-b9ca-49ac-8898-c23cf337f959-kube-api-access-4gtx5\") pod \"test-curl-11080-33742e\" (UID: \"a33c8ea3-b9ca-49ac-8898-c23cf337f959\") " pod="opendatahub/test-curl-11080-33742e"
Oct 30 19:57:10.064442 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:10.064413    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gtx5\" (UniqueName: \"kubernetes.io/projected/a33c8ea3-b9ca-49ac-8898-c23cf337f959-kube-api-access-4gtx5\") pod \"test-curl-11080-33742e\" (UID: \"a33c8ea3-b9ca-49ac-8898-c23cf337f959\") " pod="opendatahub/test-curl-11080-33742e"
Oct 30 19:57:10.077677 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:10.077653    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gtx5\" (UniqueName: \"kubernetes.io/projected/a33c8ea3-b9ca-49ac-8898-c23cf337f959-kube-api-access-4gtx5\") pod \"test-curl-11080-33742e\" (UID: \"a33c8ea3-b9ca-49ac-8898-c23cf337f959\") " pod="opendatahub/test-curl-11080-33742e"
Oct 30 19:57:10.183493 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:10.183421    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-33742e"
Oct 30 19:57:10.466213 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:10.466187    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11080-33742e"]
Oct 30 19:57:10.467897 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:57:10.467874    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda33c8ea3_b9ca_49ac_8898_c23cf337f959.slice/crio-149a9e055dbd5c2b560b3a918d795e7cf96a0b412f68be7e14553680c6e72aef WatchSource:0}: Error finding container 149a9e055dbd5c2b560b3a918d795e7cf96a0b412f68be7e14553680c6e72aef: Status 404 returned error can't find the container with id 149a9e055dbd5c2b560b3a918d795e7cf96a0b412f68be7e14553680c6e72aef
Oct 30 19:57:10.829922 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:10.829882    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11080-33742e" event={"ID":"a33c8ea3-b9ca-49ac-8898-c23cf337f959","Type":"ContainerStarted","Data":"3fa7fba30e0175faa292f6b33a90374dbf69e0e5191b191d76ea2692f0928033"}
Oct 30 19:57:10.830418 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:10.829931    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11080-33742e" event={"ID":"a33c8ea3-b9ca-49ac-8898-c23cf337f959","Type":"ContainerStarted","Data":"149a9e055dbd5c2b560b3a918d795e7cf96a0b412f68be7e14553680c6e72aef"}
Oct 30 19:57:10.830418 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:10.830097    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11080-1f54c8" podUID="abc54d37-ba6e-4915-a2d9-83970ec6ae30" containerName="test-curl-11080-1f54c8" containerID="cri-o://393226e8cbf8d9c4768126719d6bdf3e7f01a6e0a6d29d6e0372415d141a9225" gracePeriod=2
Oct 30 19:57:10.858201 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:10.858139    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11080-33742e" podStartSLOduration=1.8581268359999998 podStartE2EDuration="1.858126836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:10.85568742 +0000 UTC m=+1585.439957038" watchObservedRunningTime="2026-10-30 19:57:10.858126836 +0000 UTC m=+1585.442396453"
Oct 30 19:57:10.859514 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:57:10.859474    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11080-1f54c8\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="abc54d37-ba6e-4915-a2d9-83970ec6ae30" pod="opendatahub/test-curl-11080-1f54c8"
Oct 30 19:57:11.633261 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:11.633217    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11080-33742e"]
Oct 30 19:57:11.638832 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:11.638795    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11080-33742e"]
Oct 30 19:57:11.839211 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:11.839153    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11080-5272a4"]
Oct 30 19:57:11.840219 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:11.840192    5408 cpu_manager.go:414] "containerMap: removing container" podUID="a33c8ea3-b9ca-49ac-8898-c23cf337f959" containerName="test-curl-11080-33742e"
Oct 30 19:57:11.840219 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:11.840218    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33c8ea3-b9ca-49ac-8898-c23cf337f959" containerName="test-curl-11080-33742e"
Oct 30 19:57:11.840685 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:11.840663    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33c8ea3-b9ca-49ac-8898-c23cf337f959" containerName="test-curl-11080-33742e"
Oct 30 19:57:11.855054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:11.853208    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-5272a4"
Oct 30 19:57:11.855663 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:11.855635    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11080-5272a4"]
Oct 30 19:57:11.897611 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:11.897520    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nr6r\" (UniqueName: \"kubernetes.io/projected/75661db7-4f31-4586-97d4-53a6b4673f3a-kube-api-access-4nr6r\") pod \"test-curl-11080-5272a4\" (UID: \"75661db7-4f31-4586-97d4-53a6b4673f3a\") " pod="opendatahub/test-curl-11080-5272a4"
Oct 30 19:57:11.999877 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:11.999846    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nr6r\" (UniqueName: \"kubernetes.io/projected/75661db7-4f31-4586-97d4-53a6b4673f3a-kube-api-access-4nr6r\") pod \"test-curl-11080-5272a4\" (UID: \"75661db7-4f31-4586-97d4-53a6b4673f3a\") " pod="opendatahub/test-curl-11080-5272a4"
Oct 30 19:57:12.029529 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:12.029495    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nr6r\" (UniqueName: \"kubernetes.io/projected/75661db7-4f31-4586-97d4-53a6b4673f3a-kube-api-access-4nr6r\") pod \"test-curl-11080-5272a4\" (UID: \"75661db7-4f31-4586-97d4-53a6b4673f3a\") " pod="opendatahub/test-curl-11080-5272a4"
Oct 30 19:57:12.171073 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:12.170983    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-5272a4"
Oct 30 19:57:12.483811 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:12.483784    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11080-5272a4"]
Oct 30 19:57:12.486233 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:57:12.486204    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75661db7_4f31_4586_97d4_53a6b4673f3a.slice/crio-9ec317904db3337563770e06bc1af016bf960fe30f2eefbcedd4b5374096eb93 WatchSource:0}: Error finding container 9ec317904db3337563770e06bc1af016bf960fe30f2eefbcedd4b5374096eb93: Status 404 returned error can't find the container with id 9ec317904db3337563770e06bc1af016bf960fe30f2eefbcedd4b5374096eb93
Oct 30 19:57:12.861739 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:12.861707    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11080-5272a4" event={"ID":"75661db7-4f31-4586-97d4-53a6b4673f3a","Type":"ContainerStarted","Data":"65bb01f69a58ba538f51c95f32732f51d719fa5136b8caee1b8686e932163546"}
Oct 30 19:57:12.862164 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:12.861756    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11080-5272a4" event={"ID":"75661db7-4f31-4586-97d4-53a6b4673f3a","Type":"ContainerStarted","Data":"9ec317904db3337563770e06bc1af016bf960fe30f2eefbcedd4b5374096eb93"}
Oct 30 19:57:12.862164 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:12.861810    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11080-33742e" podUID="a33c8ea3-b9ca-49ac-8898-c23cf337f959" containerName="test-curl-11080-33742e" containerID="cri-o://3fa7fba30e0175faa292f6b33a90374dbf69e0e5191b191d76ea2692f0928033" gracePeriod=2
Oct 30 19:57:12.886651 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:12.886605    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11080-5272a4" podStartSLOduration=1.886591254 podStartE2EDuration="1.886591254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:12.885092605 +0000 UTC m=+1587.469362223" watchObservedRunningTime="2026-10-30 19:57:12.886591254 +0000 UTC m=+1587.470860875"
Oct 30 19:57:12.888175 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:57:12.888147    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11080-33742e\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="a33c8ea3-b9ca-49ac-8898-c23cf337f959" pod="opendatahub/test-curl-11080-33742e"
Oct 30 19:57:12.891063 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:57:12.891039    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11080-33742e\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="a33c8ea3-b9ca-49ac-8898-c23cf337f959" pod="opendatahub/test-curl-11080-33742e"
Oct 30 19:57:13.010549 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:13.010526    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-1f54c8"
Oct 30 19:57:13.013488 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:57:13.013462    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11080-33742e\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="a33c8ea3-b9ca-49ac-8898-c23cf337f959" pod="opendatahub/test-curl-11080-33742e"
Oct 30 19:57:13.116339 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:13.116267    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc54d37-ba6e-4915-a2d9-83970ec6ae30-kube-api-access-8bmhq\" (UniqueName: \"kubernetes.io/projected/abc54d37-ba6e-4915-a2d9-83970ec6ae30-kube-api-access-8bmhq\") pod \"abc54d37-ba6e-4915-a2d9-83970ec6ae30\" (UID: \"abc54d37-ba6e-4915-a2d9-83970ec6ae30\") "
Oct 30 19:57:13.118363 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:13.118336    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc54d37-ba6e-4915-a2d9-83970ec6ae30-kube-api-access-8bmhq" pod "abc54d37-ba6e-4915-a2d9-83970ec6ae30" (UID: "abc54d37-ba6e-4915-a2d9-83970ec6ae30"). InnerVolumeSpecName "kube-api-access-8bmhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:13.238699 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:13.238659    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bmhq\" (UniqueName: \"kubernetes.io/projected/abc54d37-ba6e-4915-a2d9-83970ec6ae30-kube-api-access-8bmhq\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:13.532889 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:13.532852    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc54d37-ba6e-4915-a2d9-83970ec6ae30" path="/var/lib/kubelet/pods/abc54d37-ba6e-4915-a2d9-83970ec6ae30/volumes"
Oct 30 19:57:13.624906 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:13.624878    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9"]
Oct 30 19:57:13.629274 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:13.629214    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9"
Oct 30 19:57:13.650864 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:13.650844    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-b0d13a\""
Oct 30 19:57:13.665274 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:13.665239    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9"]
Oct 30 19:57:13.700730 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:13.700705    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-b0d13a-dockercfg-bfmgl\""
Oct 30 19:57:13.746208 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:13.746176    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7bf362b-a122-4417-813a-1c1f30c05297-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9\" (UID: \"c7bf362b-a122-4417-813a-1c1f30c05297\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9"
Oct 30 19:57:13.746360 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:13.746311    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-654jk\" (UniqueName: \"kubernetes.io/projected/c7bf362b-a122-4417-813a-1c1f30c05297-kube-api-access-654jk\") pod \"payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9\" (UID: \"c7bf362b-a122-4417-813a-1c1f30c05297\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9"
Oct 30 19:57:13.853381 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:13.853278    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7bf362b-a122-4417-813a-1c1f30c05297-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9\" (UID: \"c7bf362b-a122-4417-813a-1c1f30c05297\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9"
Oct 30 19:57:13.863318 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:13.863278    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-654jk\" (UniqueName: \"kubernetes.io/projected/c7bf362b-a122-4417-813a-1c1f30c05297-kube-api-access-654jk\") pod \"payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9\" (UID: \"c7bf362b-a122-4417-813a-1c1f30c05297\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9"
Oct 30 19:57:13.874058 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:13.865351    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7bf362b-a122-4417-813a-1c1f30c05297-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9\" (UID: \"c7bf362b-a122-4417-813a-1c1f30c05297\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9"
Oct 30 19:57:14.003381 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.003349    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11080-5272a4"]
Oct 30 19:57:14.058052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.042770    5408 generic.go:356] "Generic (PLEG): container finished" podID="abc54d37-ba6e-4915-a2d9-83970ec6ae30" containerID="393226e8cbf8d9c4768126719d6bdf3e7f01a6e0a6d29d6e0372415d141a9225" exitCode=137
Oct 30 19:57:14.058052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.046172    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-1f54c8"
Oct 30 19:57:14.058052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.052586    5408 scope.go:122] "RemoveContainer" containerID="393226e8cbf8d9c4768126719d6bdf3e7f01a6e0a6d29d6e0372415d141a9225"
Oct 30 19:57:14.149129 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.144927    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-654jk\" (UniqueName: \"kubernetes.io/projected/c7bf362b-a122-4417-813a-1c1f30c05297-kube-api-access-654jk\") pod \"payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9\" (UID: \"c7bf362b-a122-4417-813a-1c1f30c05297\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9"
Oct 30 19:57:14.170089 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.156319    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11080-5272a4"]
Oct 30 19:57:14.184474 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.184437    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11080-e75ada"]
Oct 30 19:57:14.185951 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.185924    5408 cpu_manager.go:414] "containerMap: removing container" podUID="75661db7-4f31-4586-97d4-53a6b4673f3a" containerName="test-curl-11080-5272a4"
Oct 30 19:57:14.186200 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.186179    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75661db7-4f31-4586-97d4-53a6b4673f3a" containerName="test-curl-11080-5272a4"
Oct 30 19:57:14.186577 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.186549    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="75661db7-4f31-4586-97d4-53a6b4673f3a" containerName="test-curl-11080-5272a4"
Oct 30 19:57:14.189270 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.189241    5408 scope.go:122] "RemoveContainer" containerID="393226e8cbf8d9c4768126719d6bdf3e7f01a6e0a6d29d6e0372415d141a9225"
Oct 30 19:57:14.189727 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:57:14.189701    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"393226e8cbf8d9c4768126719d6bdf3e7f01a6e0a6d29d6e0372415d141a9225\": container with ID starting with 393226e8cbf8d9c4768126719d6bdf3e7f01a6e0a6d29d6e0372415d141a9225 not found: ID does not exist" containerID="393226e8cbf8d9c4768126719d6bdf3e7f01a6e0a6d29d6e0372415d141a9225"
Oct 30 19:57:14.197904 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.197866    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393226e8cbf8d9c4768126719d6bdf3e7f01a6e0a6d29d6e0372415d141a9225"} err="failed to get container status \"393226e8cbf8d9c4768126719d6bdf3e7f01a6e0a6d29d6e0372415d141a9225\": rpc error: code = NotFound desc = could not find container \"393226e8cbf8d9c4768126719d6bdf3e7f01a6e0a6d29d6e0372415d141a9225\": container with ID starting with 393226e8cbf8d9c4768126719d6bdf3e7f01a6e0a6d29d6e0372415d141a9225 not found: ID does not exist"
Oct 30 19:57:14.198134 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.192707    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-e75ada"
Oct 30 19:57:14.213233 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.213087    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11080-e75ada"]
Oct 30 19:57:14.245171 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.245143    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9"
Oct 30 19:57:14.268337 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.268304    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h566z\" (UniqueName: \"kubernetes.io/projected/89420585-9de2-4bc9-a57f-dc001ac9a48d-kube-api-access-h566z\") pod \"test-curl-11080-e75ada\" (UID: \"89420585-9de2-4bc9-a57f-dc001ac9a48d\") " pod="opendatahub/test-curl-11080-e75ada"
Oct 30 19:57:14.370292 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.370258    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h566z\" (UniqueName: \"kubernetes.io/projected/89420585-9de2-4bc9-a57f-dc001ac9a48d-kube-api-access-h566z\") pod \"test-curl-11080-e75ada\" (UID: \"89420585-9de2-4bc9-a57f-dc001ac9a48d\") " pod="opendatahub/test-curl-11080-e75ada"
Oct 30 19:57:14.386485 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.386458    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h566z\" (UniqueName: \"kubernetes.io/projected/89420585-9de2-4bc9-a57f-dc001ac9a48d-kube-api-access-h566z\") pod \"test-curl-11080-e75ada\" (UID: \"89420585-9de2-4bc9-a57f-dc001ac9a48d\") " pod="opendatahub/test-curl-11080-e75ada"
Oct 30 19:57:14.537570 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.537526    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-e75ada"
Oct 30 19:57:14.551613 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:14.550023    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9"]
Oct 30 19:57:15.068901 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:15.068863    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9" event={"ID":"c7bf362b-a122-4417-813a-1c1f30c05297","Type":"ContainerStarted","Data":"16f33a176cefd41b0b68e5fb1ffcddec8f52727a196325fd564e219e00eee2f7"}
Oct 30 19:57:15.069840 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:15.069817    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11080-5272a4" podUID="75661db7-4f31-4586-97d4-53a6b4673f3a" containerName="test-curl-11080-5272a4" containerID="cri-o://65bb01f69a58ba538f51c95f32732f51d719fa5136b8caee1b8686e932163546" gracePeriod=2
Oct 30 19:57:15.073693 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:57:15.073669    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11080-5272a4\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="75661db7-4f31-4586-97d4-53a6b4673f3a" pod="opendatahub/test-curl-11080-5272a4"
Oct 30 19:57:15.229174 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:15.229154    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-33742e"
Oct 30 19:57:15.232060 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:57:15.232010    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11080-5272a4\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="75661db7-4f31-4586-97d4-53a6b4673f3a" pod="opendatahub/test-curl-11080-5272a4"
Oct 30 19:57:15.241111 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:15.241091    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11080-e75ada"]
Oct 30 19:57:15.242989 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:57:15.242964    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89420585_9de2_4bc9_a57f_dc001ac9a48d.slice/crio-a01443554fbff669eabacad86a73b8081f30ac098d13ea0132236ae446f31c47 WatchSource:0}: Error finding container a01443554fbff669eabacad86a73b8081f30ac098d13ea0132236ae446f31c47: Status 404 returned error can't find the container with id a01443554fbff669eabacad86a73b8081f30ac098d13ea0132236ae446f31c47
Oct 30 19:57:15.316470 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:15.316444    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33c8ea3-b9ca-49ac-8898-c23cf337f959-kube-api-access-4gtx5\" (UniqueName: \"kubernetes.io/projected/a33c8ea3-b9ca-49ac-8898-c23cf337f959-kube-api-access-4gtx5\") pod \"a33c8ea3-b9ca-49ac-8898-c23cf337f959\" (UID: \"a33c8ea3-b9ca-49ac-8898-c23cf337f959\") "
Oct 30 19:57:15.318759 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:15.318689    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33c8ea3-b9ca-49ac-8898-c23cf337f959-kube-api-access-4gtx5" pod "a33c8ea3-b9ca-49ac-8898-c23cf337f959" (UID: "a33c8ea3-b9ca-49ac-8898-c23cf337f959"). InnerVolumeSpecName "kube-api-access-4gtx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:15.419233 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:15.419197    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gtx5\" (UniqueName: \"kubernetes.io/projected/a33c8ea3-b9ca-49ac-8898-c23cf337f959-kube-api-access-4gtx5\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:15.532194 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:57:15.532161    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11080-5272a4\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="75661db7-4f31-4586-97d4-53a6b4673f3a" pod="opendatahub/test-curl-11080-5272a4"
Oct 30 19:57:15.533854 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:15.533835    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a33c8ea3-b9ca-49ac-8898-c23cf337f959" path="/var/lib/kubelet/pods/a33c8ea3-b9ca-49ac-8898-c23cf337f959/volumes"
Oct 30 19:57:16.124987 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:16.123495    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9" event={"ID":"c7bf362b-a122-4417-813a-1c1f30c05297","Type":"ContainerStarted","Data":"044e23d5a4340c49cbe529d7f2241020a33be6ec3068162ce04592a3f837081d"}
Oct 30 19:57:16.125573 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:16.125386    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9"
Oct 30 19:57:16.131222 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:16.127015    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11080-e75ada" event={"ID":"89420585-9de2-4bc9-a57f-dc001ac9a48d","Type":"ContainerStarted","Data":"08681165d645f60503374b159377e39801a470512e40369a5d00ded1768a8e4f"}
Oct 30 19:57:16.131222 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:16.127102    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11080-e75ada" event={"ID":"89420585-9de2-4bc9-a57f-dc001ac9a48d","Type":"ContainerStarted","Data":"a01443554fbff669eabacad86a73b8081f30ac098d13ea0132236ae446f31c47"}
Oct 30 19:57:16.141681 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:16.140086    5408 generic.go:356] "Generic (PLEG): container finished" podID="a33c8ea3-b9ca-49ac-8898-c23cf337f959" containerID="3fa7fba30e0175faa292f6b33a90374dbf69e0e5191b191d76ea2692f0928033" exitCode=137
Oct 30 19:57:16.141681 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:16.140203    5408 scope.go:122] "RemoveContainer" containerID="3fa7fba30e0175faa292f6b33a90374dbf69e0e5191b191d76ea2692f0928033"
Oct 30 19:57:16.141681 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:16.140345    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-33742e"
Oct 30 19:57:16.153108 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:16.153092    5408 scope.go:122] "RemoveContainer" containerID="3fa7fba30e0175faa292f6b33a90374dbf69e0e5191b191d76ea2692f0928033"
Oct 30 19:57:16.153376 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:57:16.153357    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa7fba30e0175faa292f6b33a90374dbf69e0e5191b191d76ea2692f0928033\": container with ID starting with 3fa7fba30e0175faa292f6b33a90374dbf69e0e5191b191d76ea2692f0928033 not found: ID does not exist" containerID="3fa7fba30e0175faa292f6b33a90374dbf69e0e5191b191d76ea2692f0928033"
Oct 30 19:57:16.153461 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:16.153383    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa7fba30e0175faa292f6b33a90374dbf69e0e5191b191d76ea2692f0928033"} err="failed to get container status \"3fa7fba30e0175faa292f6b33a90374dbf69e0e5191b191d76ea2692f0928033\": rpc error: code = NotFound desc = could not find container \"3fa7fba30e0175faa292f6b33a90374dbf69e0e5191b191d76ea2692f0928033\": container with ID starting with 3fa7fba30e0175faa292f6b33a90374dbf69e0e5191b191d76ea2692f0928033 not found: ID does not exist"
Oct 30 19:57:16.158404 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:16.158365    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9" podStartSLOduration=3.158354273 podStartE2EDuration="3.158354273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:16.156138394 +0000 UTC m=+1590.740408011" watchObservedRunningTime="2026-10-30 19:57:16.158354273 +0000 UTC m=+1590.742623897"
Oct 30 19:57:16.175278 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:16.175239    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11080-e75ada" podStartSLOduration=2.175228452 podStartE2EDuration="2.175228452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:16.172732542 +0000 UTC m=+1590.757002161" watchObservedRunningTime="2026-10-30 19:57:16.175228452 +0000 UTC m=+1590.759498068"
Oct 30 19:57:16.915600 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:16.915569    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11080-e75ada"]
Oct 30 19:57:16.921385 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:16.921344    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11080-e75ada"]
Oct 30 19:57:17.150906 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:17.150876    5408 generic.go:356] "Generic (PLEG): container finished" podID="75661db7-4f31-4586-97d4-53a6b4673f3a" containerID="65bb01f69a58ba538f51c95f32732f51d719fa5136b8caee1b8686e932163546" exitCode=137
Oct 30 19:57:17.281594 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:17.281572    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-5272a4"
Oct 30 19:57:17.373430 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:17.373397    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75661db7-4f31-4586-97d4-53a6b4673f3a-kube-api-access-4nr6r\" (UniqueName: \"kubernetes.io/projected/75661db7-4f31-4586-97d4-53a6b4673f3a-kube-api-access-4nr6r\") pod \"75661db7-4f31-4586-97d4-53a6b4673f3a\" (UID: \"75661db7-4f31-4586-97d4-53a6b4673f3a\") "
Oct 30 19:57:17.375457 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:17.375432    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75661db7-4f31-4586-97d4-53a6b4673f3a-kube-api-access-4nr6r" pod "75661db7-4f31-4586-97d4-53a6b4673f3a" (UID: "75661db7-4f31-4586-97d4-53a6b4673f3a"). InnerVolumeSpecName "kube-api-access-4nr6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:17.475632 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:17.475557    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nr6r\" (UniqueName: \"kubernetes.io/projected/75661db7-4f31-4586-97d4-53a6b4673f3a-kube-api-access-4nr6r\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:17.543945 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:17.543915    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75661db7-4f31-4586-97d4-53a6b4673f3a" path="/var/lib/kubelet/pods/75661db7-4f31-4586-97d4-53a6b4673f3a/volumes"
Oct 30 19:57:18.160219 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:18.160183    5408 scope.go:122] "RemoveContainer" containerID="65bb01f69a58ba538f51c95f32732f51d719fa5136b8caee1b8686e932163546"
Oct 30 19:57:18.160709 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:18.160347    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-5272a4"
Oct 30 19:57:18.160709 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:18.160667    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11080-e75ada" podUID="89420585-9de2-4bc9-a57f-dc001ac9a48d" containerName="test-curl-11080-e75ada" containerID="cri-o://08681165d645f60503374b159377e39801a470512e40369a5d00ded1768a8e4f" gracePeriod=2
Oct 30 19:57:20.604354 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:20.604323    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-e75ada"
Oct 30 19:57:20.694088 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:20.694062    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89420585-9de2-4bc9-a57f-dc001ac9a48d-kube-api-access-h566z\" (UniqueName: \"kubernetes.io/projected/89420585-9de2-4bc9-a57f-dc001ac9a48d-kube-api-access-h566z\") pod \"89420585-9de2-4bc9-a57f-dc001ac9a48d\" (UID: \"89420585-9de2-4bc9-a57f-dc001ac9a48d\") "
Oct 30 19:57:20.696111 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:20.696082    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89420585-9de2-4bc9-a57f-dc001ac9a48d-kube-api-access-h566z" pod "89420585-9de2-4bc9-a57f-dc001ac9a48d" (UID: "89420585-9de2-4bc9-a57f-dc001ac9a48d"). InnerVolumeSpecName "kube-api-access-h566z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:20.796590 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:20.796516    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h566z\" (UniqueName: \"kubernetes.io/projected/89420585-9de2-4bc9-a57f-dc001ac9a48d-kube-api-access-h566z\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:21.270917 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:21.270880    5408 generic.go:356] "Generic (PLEG): container finished" podID="89420585-9de2-4bc9-a57f-dc001ac9a48d" containerID="08681165d645f60503374b159377e39801a470512e40369a5d00ded1768a8e4f" exitCode=137
Oct 30 19:57:21.271157 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:21.270936    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-e75ada"
Oct 30 19:57:21.271157 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:21.270978    5408 scope.go:122] "RemoveContainer" containerID="08681165d645f60503374b159377e39801a470512e40369a5d00ded1768a8e4f"
Oct 30 19:57:21.288731 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:21.288679    5408 scope.go:122] "RemoveContainer" containerID="08681165d645f60503374b159377e39801a470512e40369a5d00ded1768a8e4f"
Oct 30 19:57:21.288965 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:57:21.288938    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08681165d645f60503374b159377e39801a470512e40369a5d00ded1768a8e4f\": container with ID starting with 08681165d645f60503374b159377e39801a470512e40369a5d00ded1768a8e4f not found: ID does not exist" containerID="08681165d645f60503374b159377e39801a470512e40369a5d00ded1768a8e4f"
Oct 30 19:57:21.289071 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:21.288973    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08681165d645f60503374b159377e39801a470512e40369a5d00ded1768a8e4f"} err="failed to get container status \"08681165d645f60503374b159377e39801a470512e40369a5d00ded1768a8e4f\": rpc error: code = NotFound desc = could not find container \"08681165d645f60503374b159377e39801a470512e40369a5d00ded1768a8e4f\": container with ID starting with 08681165d645f60503374b159377e39801a470512e40369a5d00ded1768a8e4f not found: ID does not exist"
Oct 30 19:57:21.292005 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:57:21.291981    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11080-e75ada\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="89420585-9de2-4bc9-a57f-dc001ac9a48d" pod="opendatahub/test-curl-11080-e75ada"
Oct 30 19:57:21.532664 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:21.532590    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89420585-9de2-4bc9-a57f-dc001ac9a48d" path="/var/lib/kubelet/pods/89420585-9de2-4bc9-a57f-dc001ac9a48d/volumes"
Oct 30 19:57:23.714100 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:23.714065    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"]
Oct 30 19:57:23.715108 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:23.715087    5408 cpu_manager.go:414] "containerMap: removing container" podUID="89420585-9de2-4bc9-a57f-dc001ac9a48d" containerName="test-curl-11080-e75ada"
Oct 30 19:57:23.715195 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:23.715109    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89420585-9de2-4bc9-a57f-dc001ac9a48d" containerName="test-curl-11080-e75ada"
Oct 30 19:57:23.715288 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:23.715265    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="89420585-9de2-4bc9-a57f-dc001ac9a48d" containerName="test-curl-11080-e75ada"
Oct 30 19:57:23.720776 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:23.720754    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 19:57:23.730048 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:23.728976    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-b0d13a-serving-cert\""
Oct 30 19:57:23.739915 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:23.739888    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"]
Oct 30 19:57:23.880062 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:23.871257    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4fd624-deaf-4da1-9183-e83b29f52d64-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9\" (UID: \"eb4fd624-deaf-4da1-9183-e83b29f52d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 19:57:23.880062 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:23.871310    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4fd624-deaf-4da1-9183-e83b29f52d64-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9\" (UID: \"eb4fd624-deaf-4da1-9183-e83b29f52d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 19:57:23.880062 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:23.871367    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4fd624-deaf-4da1-9183-e83b29f52d64-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9\" (UID: \"eb4fd624-deaf-4da1-9183-e83b29f52d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 19:57:23.880062 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:23.871644    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n649j\" (UniqueName: \"kubernetes.io/projected/eb4fd624-deaf-4da1-9183-e83b29f52d64-kube-api-access-n649j\") pod \"maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9\" (UID: \"eb4fd624-deaf-4da1-9183-e83b29f52d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 19:57:23.992113 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:23.984561    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n649j\" (UniqueName: \"kubernetes.io/projected/eb4fd624-deaf-4da1-9183-e83b29f52d64-kube-api-access-n649j\") pod \"maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9\" (UID: \"eb4fd624-deaf-4da1-9183-e83b29f52d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 19:57:23.992113 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:23.984725    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4fd624-deaf-4da1-9183-e83b29f52d64-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9\" (UID: \"eb4fd624-deaf-4da1-9183-e83b29f52d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 19:57:23.992113 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:23.984762    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4fd624-deaf-4da1-9183-e83b29f52d64-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9\" (UID: \"eb4fd624-deaf-4da1-9183-e83b29f52d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 19:57:23.992113 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:23.984820    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4fd624-deaf-4da1-9183-e83b29f52d64-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9\" (UID: \"eb4fd624-deaf-4da1-9183-e83b29f52d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 19:57:23.992113 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:23.986087    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4fd624-deaf-4da1-9183-e83b29f52d64-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9\" (UID: \"eb4fd624-deaf-4da1-9183-e83b29f52d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 19:57:23.992113 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:57:23.986515    5408 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-serving-cert: secret "maas-api-e2e-shared-a-w0-b0d13a-serving-cert" not found
Oct 30 19:57:23.992113 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:57:23.986693    5408 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb4fd624-deaf-4da1-9183-e83b29f52d64-maas-api-tls podName:eb4fd624-deaf-4da1-9183-e83b29f52d64 nodeName:}" failed. No retries permitted until 2026-10-30 19:57:24.486670478 +0000 UTC m=+1599.070940112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb4fd624-deaf-4da1-9183-e83b29f52d64-maas-api-tls") pod "maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9" (UID: "eb4fd624-deaf-4da1-9183-e83b29f52d64") : secret "maas-api-e2e-shared-a-w0-b0d13a-serving-cert" not found
Oct 30 19:57:24.004990 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:24.000362    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4fd624-deaf-4da1-9183-e83b29f52d64-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9\" (UID: \"eb4fd624-deaf-4da1-9183-e83b29f52d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 19:57:24.022479 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:24.022409    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n649j\" (UniqueName: \"kubernetes.io/projected/eb4fd624-deaf-4da1-9183-e83b29f52d64-kube-api-access-n649j\") pod \"maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9\" (UID: \"eb4fd624-deaf-4da1-9183-e83b29f52d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 19:57:24.517384 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:24.517341    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4fd624-deaf-4da1-9183-e83b29f52d64-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9\" (UID: \"eb4fd624-deaf-4da1-9183-e83b29f52d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 19:57:24.626695 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:24.621636    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4fd624-deaf-4da1-9183-e83b29f52d64-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9\" (UID: \"eb4fd624-deaf-4da1-9183-e83b29f52d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 19:57:24.671222 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:24.656292    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 19:57:25.365057 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:57:25.364967    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4fd624_deaf_4da1_9183_e83b29f52d64.slice/crio-56dc46a4098cedd974998de82326b8e3dd76a52ecec84e3c6150892a707326c3 WatchSource:0}: Error finding container 56dc46a4098cedd974998de82326b8e3dd76a52ecec84e3c6150892a707326c3: Status 404 returned error can't find the container with id 56dc46a4098cedd974998de82326b8e3dd76a52ecec84e3c6150892a707326c3
Oct 30 19:57:25.371322 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:25.371297    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"]
Oct 30 19:57:26.310089 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:26.310055    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9" event={"ID":"eb4fd624-deaf-4da1-9183-e83b29f52d64","Type":"ContainerStarted","Data":"1c55605f8efa91f700757e87e914b4d5cb65af67a8a3015a59c44e667b80f886"}
Oct 30 19:57:26.310246 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:26.310144    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9" event={"ID":"eb4fd624-deaf-4da1-9183-e83b29f52d64","Type":"ContainerStarted","Data":"56dc46a4098cedd974998de82326b8e3dd76a52ecec84e3c6150892a707326c3"}
Oct 30 19:57:26.310246 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:26.310168    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 19:57:26.343245 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:26.343198    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9" podStartSLOduration=2.919612237 podStartE2EDuration="3.343183682s" totalImagesPullingTime="423.571445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:26.339520205 +0000 UTC m=+1600.923789822" watchObservedRunningTime="2026-10-30 19:57:26.343183682 +0000 UTC m=+1600.927453327"
Oct 30 19:57:28.162952 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:28.162922    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9"
Oct 30 19:57:32.322276 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:32.322249    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 19:57:33.697975 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:33.697938    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"]
Oct 30 19:57:33.705114 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:33.705087    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"
Oct 30 19:57:33.705350 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:33.705326    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"]
Oct 30 19:57:33.709458 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:33.709434    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ae43d70e-serving-cert\""
Oct 30 19:57:33.867583 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:33.867535    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2a68366-457b-4389-ac39-8c480e36abe3-maas-api-tls\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk\" (UID: \"a2a68366-457b-4389-ac39-8c480e36abe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"
Oct 30 19:57:33.874063 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:33.868139    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2q94\" (UniqueName: \"kubernetes.io/projected/a2a68366-457b-4389-ac39-8c480e36abe3-kube-api-access-c2q94\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk\" (UID: \"a2a68366-457b-4389-ac39-8c480e36abe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"
Oct 30 19:57:33.874629 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:33.874594    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a68366-457b-4389-ac39-8c480e36abe3-openshift-service-ca\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk\" (UID: \"a2a68366-457b-4389-ac39-8c480e36abe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"
Oct 30 19:57:33.874841 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:33.874822    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2a68366-457b-4389-ac39-8c480e36abe3-metrics-certs\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk\" (UID: \"a2a68366-457b-4389-ac39-8c480e36abe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"
Oct 30 19:57:33.991103 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:33.980497    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a68366-457b-4389-ac39-8c480e36abe3-openshift-service-ca\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk\" (UID: \"a2a68366-457b-4389-ac39-8c480e36abe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"
Oct 30 19:57:33.991103 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:33.980575    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2a68366-457b-4389-ac39-8c480e36abe3-metrics-certs\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk\" (UID: \"a2a68366-457b-4389-ac39-8c480e36abe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"
Oct 30 19:57:33.991103 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:33.980672    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2a68366-457b-4389-ac39-8c480e36abe3-maas-api-tls\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk\" (UID: \"a2a68366-457b-4389-ac39-8c480e36abe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"
Oct 30 19:57:33.991103 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:33.980735    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2q94\" (UniqueName: \"kubernetes.io/projected/a2a68366-457b-4389-ac39-8c480e36abe3-kube-api-access-c2q94\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk\" (UID: \"a2a68366-457b-4389-ac39-8c480e36abe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"
Oct 30 19:57:33.991103 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:33.981795    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a68366-457b-4389-ac39-8c480e36abe3-openshift-service-ca\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk\" (UID: \"a2a68366-457b-4389-ac39-8c480e36abe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"
Oct 30 19:57:34.007796 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:34.007479    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2a68366-457b-4389-ac39-8c480e36abe3-metrics-certs\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk\" (UID: \"a2a68366-457b-4389-ac39-8c480e36abe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"
Oct 30 19:57:34.008992 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:34.008917    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2a68366-457b-4389-ac39-8c480e36abe3-maas-api-tls\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk\" (UID: \"a2a68366-457b-4389-ac39-8c480e36abe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"
Oct 30 19:57:34.020324 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:34.020294    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2q94\" (UniqueName: \"kubernetes.io/projected/a2a68366-457b-4389-ac39-8c480e36abe3-kube-api-access-c2q94\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk\" (UID: \"a2a68366-457b-4389-ac39-8c480e36abe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"
Oct 30 19:57:34.324103 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:34.323355    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"
Oct 30 19:57:34.803675 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:34.803647    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"]
Oct 30 19:57:34.804655 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:57:34.804627    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a68366_457b_4389_ac39_8c480e36abe3.slice/crio-83265c670c8dbbba94780e34b1e0fd83f785e357862028c66352c1c62355d57f WatchSource:0}: Error finding container 83265c670c8dbbba94780e34b1e0fd83f785e357862028c66352c1c62355d57f: Status 404 returned error can't find the container with id 83265c670c8dbbba94780e34b1e0fd83f785e357862028c66352c1c62355d57f
Oct 30 19:57:35.469544 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:35.469428    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk" event={"ID":"a2a68366-457b-4389-ac39-8c480e36abe3","Type":"ContainerStarted","Data":"d6d9f11ca0ec79bcd08c57ae8b38758462faaa4f7edd5fec55126ea04fb51ea0"}
Oct 30 19:57:35.469818 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:35.469792    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk" event={"ID":"a2a68366-457b-4389-ac39-8c480e36abe3","Type":"ContainerStarted","Data":"83265c670c8dbbba94780e34b1e0fd83f785e357862028c66352c1c62355d57f"}
Oct 30 19:57:35.471183 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:35.471162    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"
Oct 30 19:57:35.522085 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:35.511790    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk" podStartSLOduration=2.104546012 podStartE2EDuration="2.511771137s" totalImagesPullingTime="407.225125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:35.510178251 +0000 UTC m=+1610.094447869" watchObservedRunningTime="2026-10-30 19:57:35.511771137 +0000 UTC m=+1610.096040758"
Oct 30 19:57:37.642054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:37.641134    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:37.642599 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:37.642320    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:37.642599 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:37.642431    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:37.675411 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:37.674584    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:37.689067 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:37.688994    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:37.740163 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:37.733102    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:38.040127 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:38.039162    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:38.168332 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:38.168288    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:39.663949 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:57:39.663891    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:57:42.568725 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:42.568698    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"
Oct 30 19:57:49.791076 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:49.791043    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x"]
Oct 30 19:57:49.797164 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:49.797144    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x"
Oct 30 19:57:49.803222 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:49.803200    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-3d0a78\""
Oct 30 19:57:49.821445 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:49.821415    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x"]
Oct 30 19:57:49.831097 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:49.831070    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-3d0a78-dockercfg-wq8s9\""
Oct 30 19:57:49.913086 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:49.908646    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e94e38-5327-48af-9f3d-a8541fe3ff0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x\" (UID: \"c3e94e38-5327-48af-9f3d-a8541fe3ff0b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x"
Oct 30 19:57:49.913086 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:49.908866    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4njfn\" (UniqueName: \"kubernetes.io/projected/c3e94e38-5327-48af-9f3d-a8541fe3ff0b-kube-api-access-4njfn\") pod \"payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x\" (UID: \"c3e94e38-5327-48af-9f3d-a8541fe3ff0b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x"
Oct 30 19:57:50.019052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:50.018195    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4njfn\" (UniqueName: \"kubernetes.io/projected/c3e94e38-5327-48af-9f3d-a8541fe3ff0b-kube-api-access-4njfn\") pod \"payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x\" (UID: \"c3e94e38-5327-48af-9f3d-a8541fe3ff0b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x"
Oct 30 19:57:50.019052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:50.018443    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e94e38-5327-48af-9f3d-a8541fe3ff0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x\" (UID: \"c3e94e38-5327-48af-9f3d-a8541fe3ff0b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x"
Oct 30 19:57:50.027088 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:50.019629    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e94e38-5327-48af-9f3d-a8541fe3ff0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x\" (UID: \"c3e94e38-5327-48af-9f3d-a8541fe3ff0b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x"
Oct 30 19:57:50.248458 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:50.240205    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4njfn\" (UniqueName: \"kubernetes.io/projected/c3e94e38-5327-48af-9f3d-a8541fe3ff0b-kube-api-access-4njfn\") pod \"payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x\" (UID: \"c3e94e38-5327-48af-9f3d-a8541fe3ff0b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x"
Oct 30 19:57:50.417075 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:50.414338    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x"
Oct 30 19:57:50.966015 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:50.965983    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x"]
Oct 30 19:57:50.968253 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:57:50.968227    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e94e38_5327_48af_9f3d_a8541fe3ff0b.slice/crio-464b1604c1f43018a8f4f17a79ab98125c7048a2659848c2a616552d781710de WatchSource:0}: Error finding container 464b1604c1f43018a8f4f17a79ab98125c7048a2659848c2a616552d781710de: Status 404 returned error can't find the container with id 464b1604c1f43018a8f4f17a79ab98125c7048a2659848c2a616552d781710de
Oct 30 19:57:51.795431 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:51.795396    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x" event={"ID":"c3e94e38-5327-48af-9f3d-a8541fe3ff0b","Type":"ContainerStarted","Data":"155e6ae005596d85269fcd4debcf04ccefd6e428906ec1b80d02c3a876862236"}
Oct 30 19:57:51.795621 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:51.795456    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x" event={"ID":"c3e94e38-5327-48af-9f3d-a8541fe3ff0b","Type":"ContainerStarted","Data":"464b1604c1f43018a8f4f17a79ab98125c7048a2659848c2a616552d781710de"}
Oct 30 19:57:51.795621 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:51.795484    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x"
Oct 30 19:57:51.817392 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:51.817347    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x" podStartSLOduration=2.8173335809999998 podStartE2EDuration="2.817333581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:51.813697492 +0000 UTC m=+1626.397967109" watchObservedRunningTime="2026-10-30 19:57:51.817333581 +0000 UTC m=+1626.401603217"
Oct 30 19:57:54.960925 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:54.960891    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"]
Oct 30 19:57:54.965674 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:54.965650    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:54.981185 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:54.981165    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ae43d70e\"/\"openshift-service-ca.crt\""
Oct 30 19:57:54.982543 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:54.982504    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ae43d70e\"/\"e2e-lifecycle-model-ae43d70e-kserve-self-signed-certs\""
Oct 30 19:57:54.982801 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:54.982777    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ae43d70e\"/\"default-dockercfg-j82qx\""
Oct 30 19:57:54.982801 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:54.982489    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ae43d70e\"/\"kube-root-ca.crt\""
Oct 30 19:57:54.989947 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:54.989919    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"]
Oct 30 19:57:55.019287 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.019251    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-tmp-dir\") pod \"e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") " pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.019453 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.019293    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-home\") pod \"e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") " pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.019453 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.019407    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-dshm\") pod \"e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") " pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.019581 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.019490    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlw7g\" (UniqueName: \"kubernetes.io/projected/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-kube-api-access-rlw7g\") pod \"e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") " pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.019581 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.019569    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-model-cache\") pod \"e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") " pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.019821 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.019803    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-tls-certs\") pod \"e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") " pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.121353 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.121318    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-tls-certs\") pod \"e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") " pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.121522 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.121390    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-tmp-dir\") pod \"e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") " pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.121522 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.121417    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-home\") pod \"e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") " pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.121522 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.121452    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-dshm\") pod \"e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") " pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.121522 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.121484    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlw7g\" (UniqueName: \"kubernetes.io/projected/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-kube-api-access-rlw7g\") pod \"e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") " pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.121760 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.121537    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-model-cache\") pod \"e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") " pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.121891 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.121863    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-tmp-dir\") pod \"e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") " pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.122122 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.121936    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-home\") pod \"e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") " pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.122122 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.121956    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-model-cache\") pod \"e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") " pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.124235 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.124210    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-dshm\") pod \"e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") " pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.124342 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.124261    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-tls-certs\") pod \"e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") " pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.135101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.135073    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlw7g\" (UniqueName: \"kubernetes.io/projected/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-kube-api-access-rlw7g\") pod \"e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") " pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.289192 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.289160    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:57:55.746115 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.746087    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"]
Oct 30 19:57:55.747428 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:57:55.747385    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf96b2a7d_14e7_4386_921d_b3bb1d6ec55d.slice/crio-d4df1f2804394b4b2afbb4f03311aa1d27ea22e75b508646c3fbbd6c69624c02 WatchSource:0}: Error finding container d4df1f2804394b4b2afbb4f03311aa1d27ea22e75b508646c3fbbd6c69624c02: Status 404 returned error can't find the container with id d4df1f2804394b4b2afbb4f03311aa1d27ea22e75b508646c3fbbd6c69624c02
Oct 30 19:57:55.839214 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.839185    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p" event={"ID":"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d","Type":"ContainerStarted","Data":"356f79db6ea832ee9dff29d7c43557d7fe380404971f61d9984be480be2ac48e"}
Oct 30 19:57:55.839337 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.839225    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p" event={"ID":"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d","Type":"ContainerStarted","Data":"d4df1f2804394b4b2afbb4f03311aa1d27ea22e75b508646c3fbbd6c69624c02"}
Oct 30 19:57:55.863605 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:55.863559    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p" podStartSLOduration=1.863546594 podStartE2EDuration="1.863546594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:55.859248043 +0000 UTC m=+1630.443517660" watchObservedRunningTime="2026-10-30 19:57:55.863546594 +0000 UTC m=+1630.447816211"
Oct 30 19:57:56.291103 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:57:56.291073    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:58:02.822101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:02.822017    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x"
Oct 30 19:58:05.290336 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:05.290303    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:58:05.303229 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:05.303194    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:58:06.291362 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:06.291329    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 19:58:09.660961 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:58:09.660920    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:58:18.770775 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:18.770738    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh"]
Oct 30 19:58:18.776362 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:18.776339    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh"
Oct 30 19:58:18.802249 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:18.802217    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh"]
Oct 30 19:58:18.832762 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:18.832725    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce5e4fb9-3865-4f30-9978-5fb9a8600a14-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh\" (UID: \"ce5e4fb9-3865-4f30-9978-5fb9a8600a14\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh"
Oct 30 19:58:18.833769 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:18.833721    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8c8v\" (UniqueName: \"kubernetes.io/projected/ce5e4fb9-3865-4f30-9978-5fb9a8600a14-kube-api-access-b8c8v\") pod \"payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh\" (UID: \"ce5e4fb9-3865-4f30-9978-5fb9a8600a14\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh"
Oct 30 19:58:18.878699 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:18.878678    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-274922\""
Oct 30 19:58:18.950152 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:18.950104    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8c8v\" (UniqueName: \"kubernetes.io/projected/ce5e4fb9-3865-4f30-9978-5fb9a8600a14-kube-api-access-b8c8v\") pod \"payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh\" (UID: \"ce5e4fb9-3865-4f30-9978-5fb9a8600a14\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh"
Oct 30 19:58:18.950951 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:18.950925    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce5e4fb9-3865-4f30-9978-5fb9a8600a14-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh\" (UID: \"ce5e4fb9-3865-4f30-9978-5fb9a8600a14\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh"
Oct 30 19:58:18.952252 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:18.951937    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce5e4fb9-3865-4f30-9978-5fb9a8600a14-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh\" (UID: \"ce5e4fb9-3865-4f30-9978-5fb9a8600a14\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh"
Oct 30 19:58:18.988226 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:18.988190    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8c8v\" (UniqueName: \"kubernetes.io/projected/ce5e4fb9-3865-4f30-9978-5fb9a8600a14-kube-api-access-b8c8v\") pod \"payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh\" (UID: \"ce5e4fb9-3865-4f30-9978-5fb9a8600a14\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh"
Oct 30 19:58:19.094236 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.094159    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh"
Oct 30 19:58:19.363166 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.363076    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"]
Oct 30 19:58:19.385257 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.385216    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"]
Oct 30 19:58:19.385587 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.385566    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.477428 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.477400    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-b0d13a-openshift-default-dockercfg-b2mpb\""
Oct 30 19:58:19.499245 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.499209    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.499456 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.499295    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.499456 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.499387    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.499590 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.499463    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn7k8\" (UniqueName: \"kubernetes.io/projected/bdae6413-f5be-452b-a5a7-d6ca4abb9081-kube-api-access-nn7k8\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.499590 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.499487    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.499704 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.499625    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.499704 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.499678    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.499816 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.499786    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.499816 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.499810    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.600646 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.600601    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.600814 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.600680    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.600814 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.600779    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.600814 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.600807    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.600999 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.600930    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.600999 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.600971    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.601132 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.601104    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.601132 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.601108    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.601297 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.601272    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn7k8\" (UniqueName: \"kubernetes.io/projected/bdae6413-f5be-452b-a5a7-d6ca4abb9081-kube-api-access-nn7k8\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.601389 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.601316    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.601501 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.601479    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.601588 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.601502    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.601588 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.601514    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.601728 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.601708    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.603428 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.603408    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.605143 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.605123    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.628143 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.628095    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.649788 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.649766    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn7k8\" (UniqueName: \"kubernetes.io/projected/bdae6413-f5be-452b-a5a7-d6ca4abb9081-kube-api-access-nn7k8\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:19.703818 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.703794    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh"]
Oct 30 19:58:19.708964 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:58:19.708943    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5e4fb9_3865_4f30_9978_5fb9a8600a14.slice/crio-a14230449c3cec177853da7cfd8384e998a23af6e3d31e75f79e8a60056077d4 WatchSource:0}: Error finding container a14230449c3cec177853da7cfd8384e998a23af6e3d31e75f79e8a60056077d4: Status 404 returned error can't find the container with id a14230449c3cec177853da7cfd8384e998a23af6e3d31e75f79e8a60056077d4
Oct 30 19:58:19.723893 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:19.723873    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:20.066623 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:20.066591    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"]
Oct 30 19:58:20.074480 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:58:20.074455    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdae6413_f5be_452b_a5a7_d6ca4abb9081.slice/crio-4c7849fb2f2b578817fd57c1812582dd55d06b95c0dd52cd9fe674c2754b6a8d WatchSource:0}: Error finding container 4c7849fb2f2b578817fd57c1812582dd55d06b95c0dd52cd9fe674c2754b6a8d: Status 404 returned error can't find the container with id 4c7849fb2f2b578817fd57c1812582dd55d06b95c0dd52cd9fe674c2754b6a8d
Oct 30 19:58:20.077329 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:20.077293    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:58:20.077429 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:20.077362    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:58:20.077429 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:20.077389    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:58:20.281386 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:20.281327    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt" event={"ID":"bdae6413-f5be-452b-a5a7-d6ca4abb9081","Type":"ContainerStarted","Data":"9f27cdc03e3ec7fcc279e3f97fd84dde90b340224211a383ad72e68cb975209e"}
Oct 30 19:58:20.281386 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:20.281389    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt" event={"ID":"bdae6413-f5be-452b-a5a7-d6ca4abb9081","Type":"ContainerStarted","Data":"4c7849fb2f2b578817fd57c1812582dd55d06b95c0dd52cd9fe674c2754b6a8d"}
Oct 30 19:58:20.283318 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:20.283280    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh" event={"ID":"ce5e4fb9-3865-4f30-9978-5fb9a8600a14","Type":"ContainerStarted","Data":"80d09244c3ca7b77b041f87cc65063d3292e97c3b9b32fb745320d4782ddd0ee"}
Oct 30 19:58:20.283462 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:20.283332    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh" event={"ID":"ce5e4fb9-3865-4f30-9978-5fb9a8600a14","Type":"ContainerStarted","Data":"a14230449c3cec177853da7cfd8384e998a23af6e3d31e75f79e8a60056077d4"}
Oct 30 19:58:20.283462 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:20.283375    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh"
Oct 30 19:58:20.337118 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:20.334855    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt" podStartSLOduration=1.334720922 podStartE2EDuration="1.334720922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:20.322904117 +0000 UTC m=+1654.907173736" watchObservedRunningTime="2026-10-30 19:58:20.334720922 +0000 UTC m=+1654.918990539"
Oct 30 19:58:20.724998 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:20.724923    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:21.742056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:21.727835    5408 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:58:21.742056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:21.727921    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt" podUID="bdae6413-f5be-452b-a5a7-d6ca4abb9081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 19:58:22.725054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:22.724420    5408 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:58:22.725054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:22.724482    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt" podUID="bdae6413-f5be-452b-a5a7-d6ca4abb9081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 19:58:23.532056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.531789    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh" podStartSLOduration=5.531768679 podStartE2EDuration="5.531768679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:20.424265474 +0000 UTC m=+1655.008535103" watchObservedRunningTime="2026-10-30 19:58:23.531768679 +0000 UTC m=+1658.116038298"
Oct 30 19:58:23.548059 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.548000    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c"]
Oct 30 19:58:23.554898 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.554878    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c"]
Oct 30 19:58:23.555069 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.554976    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c"
Oct 30 19:58:23.564607 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.564578    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb"]
Oct 30 19:58:23.570573 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.570550    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb"
Oct 30 19:58:23.571455 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.571427    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-eaa5cd\""
Oct 30 19:58:23.604054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.587911    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb"]
Oct 30 19:58:23.604054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.596139    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-eaa5cd-dockercfg-9sd9m\""
Oct 30 19:58:23.645767 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.645717    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7fv4\" (UniqueName: \"kubernetes.io/projected/2375fc63-7566-4334-98a8-7436a6ddb6ae-kube-api-access-h7fv4\") pod \"payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c\" (UID: \"2375fc63-7566-4334-98a8-7436a6ddb6ae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c"
Oct 30 19:58:23.646007 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.645941    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94a598d-85ad-41bc-afca-a4e45dcf6f06-plugins-config-volume\") pod \"payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb\" (UID: \"f94a598d-85ad-41bc-afca-a4e45dcf6f06\") " pod="openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb"
Oct 30 19:58:23.646007 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.646003    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4nfw\" (UniqueName: \"kubernetes.io/projected/f94a598d-85ad-41bc-afca-a4e45dcf6f06-kube-api-access-d4nfw\") pod \"payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb\" (UID: \"f94a598d-85ad-41bc-afca-a4e45dcf6f06\") " pod="openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb"
Oct 30 19:58:23.664044 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.646281    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2375fc63-7566-4334-98a8-7436a6ddb6ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c\" (UID: \"2375fc63-7566-4334-98a8-7436a6ddb6ae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c"
Oct 30 19:58:23.729098 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.725512    5408 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:58:23.729098 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.725587    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt" podUID="bdae6413-f5be-452b-a5a7-d6ca4abb9081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 19:58:23.800105 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.799566    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2375fc63-7566-4334-98a8-7436a6ddb6ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c\" (UID: \"2375fc63-7566-4334-98a8-7436a6ddb6ae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c"
Oct 30 19:58:23.800105 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.799719    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7fv4\" (UniqueName: \"kubernetes.io/projected/2375fc63-7566-4334-98a8-7436a6ddb6ae-kube-api-access-h7fv4\") pod \"payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c\" (UID: \"2375fc63-7566-4334-98a8-7436a6ddb6ae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c"
Oct 30 19:58:23.800105 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.799862    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94a598d-85ad-41bc-afca-a4e45dcf6f06-plugins-config-volume\") pod \"payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb\" (UID: \"f94a598d-85ad-41bc-afca-a4e45dcf6f06\") " pod="openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb"
Oct 30 19:58:23.800105 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.799905    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4nfw\" (UniqueName: \"kubernetes.io/projected/f94a598d-85ad-41bc-afca-a4e45dcf6f06-kube-api-access-d4nfw\") pod \"payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb\" (UID: \"f94a598d-85ad-41bc-afca-a4e45dcf6f06\") " pod="openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb"
Oct 30 19:58:23.855682 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.801443    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2375fc63-7566-4334-98a8-7436a6ddb6ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c\" (UID: \"2375fc63-7566-4334-98a8-7436a6ddb6ae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c"
Oct 30 19:58:23.855682 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.802500    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94a598d-85ad-41bc-afca-a4e45dcf6f06-plugins-config-volume\") pod \"payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb\" (UID: \"f94a598d-85ad-41bc-afca-a4e45dcf6f06\") " pod="openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb"
Oct 30 19:58:23.883071 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.860109    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4nfw\" (UniqueName: \"kubernetes.io/projected/f94a598d-85ad-41bc-afca-a4e45dcf6f06-kube-api-access-d4nfw\") pod \"payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb\" (UID: \"f94a598d-85ad-41bc-afca-a4e45dcf6f06\") " pod="openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb"
Oct 30 19:58:23.911056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:23.908454    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb"
Oct 30 19:58:24.014051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:24.005945    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7fv4\" (UniqueName: \"kubernetes.io/projected/2375fc63-7566-4334-98a8-7436a6ddb6ae-kube-api-access-h7fv4\") pod \"payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c\" (UID: \"2375fc63-7566-4334-98a8-7436a6ddb6ae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c"
Oct 30 19:58:24.202270 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:24.201190    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c"
Oct 30 19:58:24.749233 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:24.748142    5408 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:58:24.749233 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:24.748200    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt" podUID="bdae6413-f5be-452b-a5a7-d6ca4abb9081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 19:58:24.757215 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:24.757152    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c"]
Oct 30 19:58:24.760909 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:24.760872    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb"]
Oct 30 19:58:25.630041 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:25.629990    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c"
Oct 30 19:58:25.630374 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:25.630332    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c" event={"ID":"2375fc63-7566-4334-98a8-7436a6ddb6ae","Type":"ContainerStarted","Data":"9e3b0505ad64f63fe1e34b3d0278247d9d771c0a38a18c69705c7986a0211eee"}
Oct 30 19:58:25.630526 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:25.630512    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb"
Oct 30 19:58:25.630665 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:25.630651    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c" event={"ID":"2375fc63-7566-4334-98a8-7436a6ddb6ae","Type":"ContainerStarted","Data":"3e9125319ab423307aeb3efa94a2318c1fffc1272947cfceb5c652f6bd5068f1"}
Oct 30 19:58:25.630786 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:25.630770    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb" event={"ID":"f94a598d-85ad-41bc-afca-a4e45dcf6f06","Type":"ContainerStarted","Data":"7195c2ce753542ab552325341190e900a77afe86d65e402b10c170f56d4ea296"}
Oct 30 19:58:25.630909 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:25.630895    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb" event={"ID":"f94a598d-85ad-41bc-afca-a4e45dcf6f06","Type":"ContainerStarted","Data":"6de6ba4e995c1d24469f6d7bdfb843805164d95e972ba54ebf6ef63f58049c47"}
Oct 30 19:58:25.668051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:25.663799    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c" podStartSLOduration=2.6637791 podStartE2EDuration="2.6637791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:25.652427176 +0000 UTC m=+1660.236696795" watchObservedRunningTime="2026-10-30 19:58:25.6637791 +0000 UTC m=+1660.248048719"
Oct 30 19:58:25.720073 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:25.715597    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb" podStartSLOduration=2.7155791750000002 podStartE2EDuration="2.715579175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:25.678938268 +0000 UTC m=+1660.263207887" watchObservedRunningTime="2026-10-30 19:58:25.715579175 +0000 UTC m=+1660.299848792"
Oct 30 19:58:25.780055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:25.778598    5408 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:58:25.780055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:25.778656    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt" podUID="bdae6413-f5be-452b-a5a7-d6ca4abb9081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 19:58:26.727434 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:26.727400    5408 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:58:26.727736 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:26.727697    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt" podUID="bdae6413-f5be-452b-a5a7-d6ca4abb9081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 19:58:27.725168 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:27.725137    5408 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:58:27.725611 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:27.725188    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt" podUID="bdae6413-f5be-452b-a5a7-d6ca4abb9081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 19:58:28.729413 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:28.729375    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:28.729905 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:28.729882    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:28.730698 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:28.730668    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 19:58:31.553413 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:31.553384    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh"
Oct 30 19:58:35.176222 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:35.176163    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"]
Oct 30 19:58:35.183459 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:35.183436    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"
Oct 30 19:58:35.197600 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:35.197576    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-274922-serving-cert\""
Oct 30 19:58:35.207988 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:35.207961    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"]
Oct 30 19:58:35.256018 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:35.255986    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm\" (UID: \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"
Oct 30 19:58:35.256207 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:35.256052    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm\" (UID: \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"
Oct 30 19:58:35.256333 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:35.256308    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5mrc\" (UniqueName: \"kubernetes.io/projected/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-kube-api-access-z5mrc\") pod \"maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm\" (UID: \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"
Oct 30 19:58:35.256458 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:35.256433    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm\" (UID: \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"
Oct 30 19:58:35.358624 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:35.358583    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5mrc\" (UniqueName: \"kubernetes.io/projected/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-kube-api-access-z5mrc\") pod \"maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm\" (UID: \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"
Oct 30 19:58:35.358798 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:35.358648    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm\" (UID: \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"
Oct 30 19:58:35.358841 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:35.358805    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm\" (UID: \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"
Oct 30 19:58:35.358841 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:35.358833    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm\" (UID: \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"
Oct 30 19:58:35.359535 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:35.359507    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm\" (UID: \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"
Oct 30 19:58:35.361213 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:35.361185    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm\" (UID: \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"
Oct 30 19:58:35.361322 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:35.361296    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm\" (UID: \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"
Oct 30 19:58:35.372070 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:35.372016    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5mrc\" (UniqueName: \"kubernetes.io/projected/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-kube-api-access-z5mrc\") pod \"maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm\" (UID: \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"
Oct 30 19:58:35.502786 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:35.502751    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"
Oct 30 19:58:36.042291 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:36.042142    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"]
Oct 30 19:58:36.640253 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:36.640224    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb"
Oct 30 19:58:36.640666 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:36.640302    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c"
Oct 30 19:58:36.835516 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:36.835476    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm" event={"ID":"bb98f97c-6cc1-4731-b3e8-e906dfb00e65","Type":"ContainerStarted","Data":"9118469e9b9bb832be8d3cea88b6f9c275905d486d5b0a424fdad585f17ffac9"}
Oct 30 19:58:36.835722 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:36.835549    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm" event={"ID":"bb98f97c-6cc1-4731-b3e8-e906dfb00e65","Type":"ContainerStarted","Data":"16cbbc852f4bcff508b4d1a4ce1d0f5168e707410d7e6fb5c2c9ca232eb8930d"}
Oct 30 19:58:36.836643 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:36.836621    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"
Oct 30 19:58:36.877641 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:36.877597    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm" podStartSLOduration=1.280412584 podStartE2EDuration="1.87758502s" totalImagesPullingTime="597.172436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:36.872897685 +0000 UTC m=+1671.457167302" watchObservedRunningTime="2026-10-30 19:58:36.87758502 +0000 UTC m=+1671.461854656"
Oct 30 19:58:38.691352 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:38.691318    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"]
Oct 30 19:58:38.695832 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:38.695811    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 19:58:38.713648 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:38.713629    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-eaa5cd-serving-cert\""
Oct 30 19:58:38.723399 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:38.723377    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"]
Oct 30 19:58:38.849054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:38.848359    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/890daa8d-efaa-423f-8d7c-172b4e6081ff-metrics-certs\") pod \"maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4\" (UID: \"890daa8d-efaa-423f-8d7c-172b4e6081ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 19:58:38.849054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:38.848478    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/890daa8d-efaa-423f-8d7c-172b4e6081ff-openshift-service-ca\") pod \"maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4\" (UID: \"890daa8d-efaa-423f-8d7c-172b4e6081ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 19:58:38.849054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:38.848531    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74ph9\" (UniqueName: \"kubernetes.io/projected/890daa8d-efaa-423f-8d7c-172b4e6081ff-kube-api-access-74ph9\") pod \"maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4\" (UID: \"890daa8d-efaa-423f-8d7c-172b4e6081ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 19:58:38.849054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:38.848734    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/890daa8d-efaa-423f-8d7c-172b4e6081ff-maas-api-tls\") pod \"maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4\" (UID: \"890daa8d-efaa-423f-8d7c-172b4e6081ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 19:58:38.952127 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:38.951375    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/890daa8d-efaa-423f-8d7c-172b4e6081ff-maas-api-tls\") pod \"maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4\" (UID: \"890daa8d-efaa-423f-8d7c-172b4e6081ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 19:58:38.952127 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:38.951643    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/890daa8d-efaa-423f-8d7c-172b4e6081ff-metrics-certs\") pod \"maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4\" (UID: \"890daa8d-efaa-423f-8d7c-172b4e6081ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 19:58:38.952127 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:38.951764    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/890daa8d-efaa-423f-8d7c-172b4e6081ff-openshift-service-ca\") pod \"maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4\" (UID: \"890daa8d-efaa-423f-8d7c-172b4e6081ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 19:58:38.952127 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:38.951833    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74ph9\" (UniqueName: \"kubernetes.io/projected/890daa8d-efaa-423f-8d7c-172b4e6081ff-kube-api-access-74ph9\") pod \"maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4\" (UID: \"890daa8d-efaa-423f-8d7c-172b4e6081ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 19:58:38.952457 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:58:38.952437    5408 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-serving-cert: secret "maas-api-e2e-api-a-eaa5cd-serving-cert" not found
Oct 30 19:58:38.952531 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:58:38.952513    5408 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/890daa8d-efaa-423f-8d7c-172b4e6081ff-maas-api-tls podName:890daa8d-efaa-423f-8d7c-172b4e6081ff nodeName:}" failed. No retries permitted until 2026-10-30 19:58:39.452490414 +0000 UTC m=+1674.036760026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/890daa8d-efaa-423f-8d7c-172b4e6081ff-maas-api-tls") pod "maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4" (UID: "890daa8d-efaa-423f-8d7c-172b4e6081ff") : secret "maas-api-e2e-api-a-eaa5cd-serving-cert" not found
Oct 30 19:58:38.953340 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:38.953317    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/890daa8d-efaa-423f-8d7c-172b4e6081ff-openshift-service-ca\") pod \"maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4\" (UID: \"890daa8d-efaa-423f-8d7c-172b4e6081ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 19:58:38.955103 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:38.955080    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/890daa8d-efaa-423f-8d7c-172b4e6081ff-metrics-certs\") pod \"maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4\" (UID: \"890daa8d-efaa-423f-8d7c-172b4e6081ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 19:58:38.971471 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:38.971447    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74ph9\" (UniqueName: \"kubernetes.io/projected/890daa8d-efaa-423f-8d7c-172b4e6081ff-kube-api-access-74ph9\") pod \"maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4\" (UID: \"890daa8d-efaa-423f-8d7c-172b4e6081ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 19:58:39.461528 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:39.461488    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/890daa8d-efaa-423f-8d7c-172b4e6081ff-maas-api-tls\") pod \"maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4\" (UID: \"890daa8d-efaa-423f-8d7c-172b4e6081ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 19:58:39.463970 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:39.463946    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/890daa8d-efaa-423f-8d7c-172b4e6081ff-maas-api-tls\") pod \"maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4\" (UID: \"890daa8d-efaa-423f-8d7c-172b4e6081ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 19:58:39.613052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:39.610233    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 19:58:39.821056 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:58:39.814556    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:58:40.137264 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:40.137231    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"]
Oct 30 19:58:40.138748 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:58:40.138720    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890daa8d_efaa_423f_8d7c_172b4e6081ff.slice/crio-16d18f4e7139ddf5e92a4e92898f9cd784a3c3dcca9b86362e670dd8457020f3 WatchSource:0}: Error finding container 16d18f4e7139ddf5e92a4e92898f9cd784a3c3dcca9b86362e670dd8457020f3: Status 404 returned error can't find the container with id 16d18f4e7139ddf5e92a4e92898f9cd784a3c3dcca9b86362e670dd8457020f3
Oct 30 19:58:40.931251 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:40.931171    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4" event={"ID":"890daa8d-efaa-423f-8d7c-172b4e6081ff","Type":"ContainerStarted","Data":"0f8b945db3af662cb17ff892767f91465945d4c5193c9dd40434337093d8e05d"}
Oct 30 19:58:40.931251 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:40.931230    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4" event={"ID":"890daa8d-efaa-423f-8d7c-172b4e6081ff","Type":"ContainerStarted","Data":"16d18f4e7139ddf5e92a4e92898f9cd784a3c3dcca9b86362e670dd8457020f3"}
Oct 30 19:58:40.931706 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:40.931394    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 19:58:40.960399 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:40.960275    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4" podStartSLOduration=2.465320993 podStartE2EDuration="2.960251379s" totalImagesPullingTime="494.930386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:40.958310855 +0000 UTC m=+1675.542580473" watchObservedRunningTime="2026-10-30 19:58:40.960251379 +0000 UTC m=+1675.544520998"
Oct 30 19:58:43.917772 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:43.917742    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"
Oct 30 19:58:46.945384 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:58:46.945337    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 19:59:04.497153 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.497117    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"]
Oct 30 19:59:04.511302 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.511275    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"]
Oct 30 19:59:04.511657 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.511570    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.556180 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.556149    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.556355 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.556204    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb9p5\" (UniqueName: \"kubernetes.io/projected/7577f068-6466-4add-8edf-691d113e8eb2-kube-api-access-gb9p5\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.556410 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.556352    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.556410 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.556397    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7577f068-6466-4add-8edf-691d113e8eb2-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.556522 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.556507    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.556557 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.556546    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7577f068-6466-4add-8edf-691d113e8eb2-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.556648 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.556628    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.556797 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.556667    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7577f068-6466-4add-8edf-691d113e8eb2-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.556797 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.556760    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.658257 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.658215    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.658443 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.658276    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7577f068-6466-4add-8edf-691d113e8eb2-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.658443 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.658366    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.658443 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.658414    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7577f068-6466-4add-8edf-691d113e8eb2-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.658618 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.658494    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.659102 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.658764    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.659102 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.658827    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb9p5\" (UniqueName: \"kubernetes.io/projected/7577f068-6466-4add-8edf-691d113e8eb2-kube-api-access-gb9p5\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.659102 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.658906    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.659102 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.658911    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.659102 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.658937    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7577f068-6466-4add-8edf-691d113e8eb2-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.660807 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.659480    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.660807 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.659857    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7577f068-6466-4add-8edf-691d113e8eb2-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.660807 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.659991    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.660807 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.660240    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.661566 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.661542    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.663341 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.663317    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7577f068-6466-4add-8edf-691d113e8eb2-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.672419 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.672393    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb9p5\" (UniqueName: \"kubernetes.io/projected/7577f068-6466-4add-8edf-691d113e8eb2-kube-api-access-gb9p5\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.680222 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.680200    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7577f068-6466-4add-8edf-691d113e8eb2-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.721274 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.721229    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6"]
Oct 30 19:59:04.728478 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.728448    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf"]
Oct 30 19:59:04.728629 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.728553    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6"
Oct 30 19:59:04.733474 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.733445    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf"
Oct 30 19:59:04.740052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.738692    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6"]
Oct 30 19:59:04.745206 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.743418    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf"]
Oct 30 19:59:04.754943 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.754918    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-ae2147\""
Oct 30 19:59:04.755107 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.755079    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-ae2147-dockercfg-hm4wq\""
Oct 30 19:59:04.837051 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.832231    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:04.929305 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.929265    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69k8w\" (UniqueName: \"kubernetes.io/projected/b623a795-c956-45ca-bf19-976ff116e46c-kube-api-access-69k8w\") pod \"payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf\" (UID: \"b623a795-c956-45ca-bf19-976ff116e46c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf"
Oct 30 19:59:04.929447 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.929361    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9lhm\" (UniqueName: \"kubernetes.io/projected/33ef1fdb-5a75-4de2-833b-ec7868412227-kube-api-access-r9lhm\") pod \"payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6\" (UID: \"33ef1fdb-5a75-4de2-833b-ec7868412227\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6"
Oct 30 19:59:04.929447 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.929414    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ef1fdb-5a75-4de2-833b-ec7868412227-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6\" (UID: \"33ef1fdb-5a75-4de2-833b-ec7868412227\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6"
Oct 30 19:59:04.933052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:04.929733    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b623a795-c956-45ca-bf19-976ff116e46c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf\" (UID: \"b623a795-c956-45ca-bf19-976ff116e46c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf"
Oct 30 19:59:05.039152 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:05.030983    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ef1fdb-5a75-4de2-833b-ec7868412227-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6\" (UID: \"33ef1fdb-5a75-4de2-833b-ec7868412227\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6"
Oct 30 19:59:05.039152 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:05.031273    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b623a795-c956-45ca-bf19-976ff116e46c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf\" (UID: \"b623a795-c956-45ca-bf19-976ff116e46c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf"
Oct 30 19:59:05.039152 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:05.031432    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69k8w\" (UniqueName: \"kubernetes.io/projected/b623a795-c956-45ca-bf19-976ff116e46c-kube-api-access-69k8w\") pod \"payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf\" (UID: \"b623a795-c956-45ca-bf19-976ff116e46c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf"
Oct 30 19:59:05.039152 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:05.031489    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9lhm\" (UniqueName: \"kubernetes.io/projected/33ef1fdb-5a75-4de2-833b-ec7868412227-kube-api-access-r9lhm\") pod \"payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6\" (UID: \"33ef1fdb-5a75-4de2-833b-ec7868412227\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6"
Oct 30 19:59:05.039152 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:05.032720    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ef1fdb-5a75-4de2-833b-ec7868412227-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6\" (UID: \"33ef1fdb-5a75-4de2-833b-ec7868412227\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6"
Oct 30 19:59:05.068377 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:05.068339    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b623a795-c956-45ca-bf19-976ff116e46c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf\" (UID: \"b623a795-c956-45ca-bf19-976ff116e46c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf"
Oct 30 19:59:05.083101 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:05.072609    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9lhm\" (UniqueName: \"kubernetes.io/projected/33ef1fdb-5a75-4de2-833b-ec7868412227-kube-api-access-r9lhm\") pod \"payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6\" (UID: \"33ef1fdb-5a75-4de2-833b-ec7868412227\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6"
Oct 30 19:59:05.111342 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:05.110006    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69k8w\" (UniqueName: \"kubernetes.io/projected/b623a795-c956-45ca-bf19-976ff116e46c-kube-api-access-69k8w\") pod \"payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf\" (UID: \"b623a795-c956-45ca-bf19-976ff116e46c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf"
Oct 30 19:59:05.398116 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:05.396513    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6"
Oct 30 19:59:05.398116 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:05.397432    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf"
Oct 30 19:59:05.642783 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:05.642722    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"]
Oct 30 19:59:05.643583 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:59:05.643555    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7577f068_6466_4add_8edf_691d113e8eb2.slice/crio-7258dcc428a853143b6d29d711f2238629cc7c5ab149a6b6a9f1c5e93145b57c WatchSource:0}: Error finding container 7258dcc428a853143b6d29d711f2238629cc7c5ab149a6b6a9f1c5e93145b57c: Status 404 returned error can't find the container with id 7258dcc428a853143b6d29d711f2238629cc7c5ab149a6b6a9f1c5e93145b57c
Oct 30 19:59:05.646686 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:05.646652    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:59:05.646794 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:05.646716    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:59:05.646794 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:05.646766    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:59:05.787633 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:05.787168    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf"]
Oct 30 19:59:05.793487 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:05.793341    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6"]
Oct 30 19:59:06.462513 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:06.462472    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp" event={"ID":"7577f068-6466-4add-8edf-691d113e8eb2","Type":"ContainerStarted","Data":"7301acfcb0c3035d3f5a59ff56d0a539d59054eb24a09e62efcc35ebc78912a8"}
Oct 30 19:59:06.462721 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:06.462533    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp" event={"ID":"7577f068-6466-4add-8edf-691d113e8eb2","Type":"ContainerStarted","Data":"7258dcc428a853143b6d29d711f2238629cc7c5ab149a6b6a9f1c5e93145b57c"}
Oct 30 19:59:06.464123 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:06.464091    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf" event={"ID":"b623a795-c956-45ca-bf19-976ff116e46c","Type":"ContainerStarted","Data":"9f74a27177e1c21edf2a3a25a9c34e8f0e4cff5134e7c81c739fabfade76b151"}
Oct 30 19:59:06.464260 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:06.464141    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf" event={"ID":"b623a795-c956-45ca-bf19-976ff116e46c","Type":"ContainerStarted","Data":"45d4c2cf1f59e9c4c35f05380287335f6fa8a37695d4649396d77ca2d4b53ab2"}
Oct 30 19:59:06.464260 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:06.464230    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf"
Oct 30 19:59:06.465620 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:06.465601    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6" event={"ID":"33ef1fdb-5a75-4de2-833b-ec7868412227","Type":"ContainerStarted","Data":"9a72686bced51c8467a6e1cfab8707f034a596b98a0092ddba2f51091f29c168"}
Oct 30 19:59:06.465725 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:06.465637    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6" event={"ID":"33ef1fdb-5a75-4de2-833b-ec7868412227","Type":"ContainerStarted","Data":"27a7b15b1b62097954e0ef23f0547fa585346ead9cdff0ed234b8d6e22ec0c6c"}
Oct 30 19:59:06.465725 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:06.465664    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6"
Oct 30 19:59:06.488122 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:06.488018    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp" podStartSLOduration=2.487999115 podStartE2EDuration="2.487999115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:06.486820281 +0000 UTC m=+1701.071089896" watchObservedRunningTime="2026-10-30 19:59:06.487999115 +0000 UTC m=+1701.072268751"
Oct 30 19:59:06.516280 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:06.516214    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf" podStartSLOduration=2.516194029 podStartE2EDuration="2.516194029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:06.506878463 +0000 UTC m=+1701.091148081" watchObservedRunningTime="2026-10-30 19:59:06.516194029 +0000 UTC m=+1701.100463649"
Oct 30 19:59:06.542045 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:06.541986    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6" podStartSLOduration=2.541970688 podStartE2EDuration="2.541970688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:06.538579422 +0000 UTC m=+1701.122849036" watchObservedRunningTime="2026-10-30 19:59:06.541970688 +0000 UTC m=+1701.126240305"
Oct 30 19:59:06.838292 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:06.838246    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:06.839863 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:06.839839    5408 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:59:06.839949 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:06.839884    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp" podUID="7577f068-6466-4add-8edf-691d113e8eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 19:59:07.838754 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:07.838706    5408 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:59:07.839169 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:07.838779    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp" podUID="7577f068-6466-4add-8edf-691d113e8eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 19:59:08.838441 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:08.838399    5408 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:59:08.838645 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:08.838468    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp" podUID="7577f068-6466-4add-8edf-691d113e8eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 19:59:09.680531 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:59:09.680490    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:59:09.847054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:09.843717    5408 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:59:09.847054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:09.843805    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp" podUID="7577f068-6466-4add-8edf-691d113e8eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 19:59:10.727056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:10.726568    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp"]
Oct 30 19:59:10.735171 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:10.735138    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp"
Oct 30 19:59:10.736487 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:10.736457    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp"]
Oct 30 19:59:10.745129 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:10.745082    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a69dfb51-e759-48f5-9a34-f1047bc4c2c2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp\" (UID: \"a69dfb51-e759-48f5-9a34-f1047bc4c2c2\") " pod="openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp"
Oct 30 19:59:10.745562 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:10.745538    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph8xg\" (UniqueName: \"kubernetes.io/projected/a69dfb51-e759-48f5-9a34-f1047bc4c2c2-kube-api-access-ph8xg\") pod \"payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp\" (UID: \"a69dfb51-e759-48f5-9a34-f1047bc4c2c2\") " pod="openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp"
Oct 30 19:59:10.751927 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:10.751906    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c4c072\""
Oct 30 19:59:10.752487 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:10.752468    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c4c072-dockercfg-cbmt4\""
Oct 30 19:59:10.842995 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:10.842960    5408 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:59:10.843822 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:10.843255    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp" podUID="7577f068-6466-4add-8edf-691d113e8eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 19:59:10.850084 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:10.848450    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph8xg\" (UniqueName: \"kubernetes.io/projected/a69dfb51-e759-48f5-9a34-f1047bc4c2c2-kube-api-access-ph8xg\") pod \"payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp\" (UID: \"a69dfb51-e759-48f5-9a34-f1047bc4c2c2\") " pod="openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp"
Oct 30 19:59:10.850084 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:10.848625    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a69dfb51-e759-48f5-9a34-f1047bc4c2c2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp\" (UID: \"a69dfb51-e759-48f5-9a34-f1047bc4c2c2\") " pod="openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp"
Oct 30 19:59:10.850084 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:10.849760    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a69dfb51-e759-48f5-9a34-f1047bc4c2c2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp\" (UID: \"a69dfb51-e759-48f5-9a34-f1047bc4c2c2\") " pod="openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp"
Oct 30 19:59:10.886067 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:10.884576    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph8xg\" (UniqueName: \"kubernetes.io/projected/a69dfb51-e759-48f5-9a34-f1047bc4c2c2-kube-api-access-ph8xg\") pod \"payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp\" (UID: \"a69dfb51-e759-48f5-9a34-f1047bc4c2c2\") " pod="openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp"
Oct 30 19:59:11.092969 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:11.092657    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp"
Oct 30 19:59:11.927068 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:11.923131    5408 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:59:11.927068 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:11.923218    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp" podUID="7577f068-6466-4add-8edf-691d113e8eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 19:59:11.964269 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:11.953220    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp"]
Oct 30 19:59:12.559909 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:12.559873    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp" event={"ID":"a69dfb51-e759-48f5-9a34-f1047bc4c2c2","Type":"ContainerStarted","Data":"899f9f4eff92120e4ecda0530fe8269e99de0751b704671095406973351495e5"}
Oct 30 19:59:12.560154 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:12.559939    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp" event={"ID":"a69dfb51-e759-48f5-9a34-f1047bc4c2c2","Type":"ContainerStarted","Data":"cb96bbb323c9b518119986d266e123f2186b2c283b1b5edd92f6f8e4f30e89fa"}
Oct 30 19:59:12.560154 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:12.559996    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp"
Oct 30 19:59:12.851220 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:12.847080    5408 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:59:12.851220 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:12.847156    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp" podUID="7577f068-6466-4add-8edf-691d113e8eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 19:59:13.864085 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:13.862352    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:13.864085 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:13.863656    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:13.864634 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:13.864498    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 19:59:13.937523 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:13.937461    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp" podStartSLOduration=3.937436357 podStartE2EDuration="3.937436357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:12.612792238 +0000 UTC m=+1707.197061857" watchObservedRunningTime="2026-10-30 19:59:13.937436357 +0000 UTC m=+1708.521705979"
Oct 30 19:59:17.507627 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:17.498989    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6"
Oct 30 19:59:17.507627 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:17.500837    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf"
Oct 30 19:59:21.608706 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:21.608670    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11080-929e15"]
Oct 30 19:59:21.615000 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:21.614979    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-929e15"
Oct 30 19:59:21.621483 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:21.620626    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xqd5g\""
Oct 30 19:59:21.623951 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:21.623918    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11080-929e15"]
Oct 30 19:59:21.652791 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:21.652755    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6wrf\" (UniqueName: \"kubernetes.io/projected/491b27e8-f44a-455e-97c9-a35fb5f2d39e-kube-api-access-w6wrf\") pod \"test-curl-11080-929e15\" (UID: \"491b27e8-f44a-455e-97c9-a35fb5f2d39e\") " pod="opendatahub/test-curl-11080-929e15"
Oct 30 19:59:21.781891 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:21.781854    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6wrf\" (UniqueName: \"kubernetes.io/projected/491b27e8-f44a-455e-97c9-a35fb5f2d39e-kube-api-access-w6wrf\") pod \"test-curl-11080-929e15\" (UID: \"491b27e8-f44a-455e-97c9-a35fb5f2d39e\") " pod="opendatahub/test-curl-11080-929e15"
Oct 30 19:59:21.819820 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:21.816515    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6wrf\" (UniqueName: \"kubernetes.io/projected/491b27e8-f44a-455e-97c9-a35fb5f2d39e-kube-api-access-w6wrf\") pod \"test-curl-11080-929e15\" (UID: \"491b27e8-f44a-455e-97c9-a35fb5f2d39e\") " pod="opendatahub/test-curl-11080-929e15"
Oct 30 19:59:21.939205 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:21.939109    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-929e15"
Oct 30 19:59:22.228938 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:22.228907    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11080-929e15"]
Oct 30 19:59:22.229753 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:59:22.229722    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod491b27e8_f44a_455e_97c9_a35fb5f2d39e.slice/crio-1696b6e004eef995134607fb215d6a53fc88a90a73e440813e6457ec4f3a199b WatchSource:0}: Error finding container 1696b6e004eef995134607fb215d6a53fc88a90a73e440813e6457ec4f3a199b: Status 404 returned error can't find the container with id 1696b6e004eef995134607fb215d6a53fc88a90a73e440813e6457ec4f3a199b
Oct 30 19:59:22.826381 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:22.826345    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11080-929e15" event={"ID":"491b27e8-f44a-455e-97c9-a35fb5f2d39e","Type":"ContainerStarted","Data":"231d1b88746a0c37a5dd2e868b1725bb90a6c3b73ec5143c5741a45aa61ed1b6"}
Oct 30 19:59:22.826816 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:22.826391    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11080-929e15" event={"ID":"491b27e8-f44a-455e-97c9-a35fb5f2d39e","Type":"ContainerStarted","Data":"1696b6e004eef995134607fb215d6a53fc88a90a73e440813e6457ec4f3a199b"}
Oct 30 19:59:22.855865 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:22.855802    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11080-929e15" podStartSLOduration=1.85578531 podStartE2EDuration="1.85578531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:22.852382864 +0000 UTC m=+1717.436652545" watchObservedRunningTime="2026-10-30 19:59:22.85578531 +0000 UTC m=+1717.440054926"
Oct 30 19:59:22.973436 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:22.973399    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"]
Oct 30 19:59:22.977872 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:22.977853    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:22.983382 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:22.983347    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dd1b04-openshift-default-dockercfg-dhzsm\""
Oct 30 19:59:22.993493 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:22.993469    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"]
Oct 30 19:59:23.001216 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.001184    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-envoy\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.001425 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.001306    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-token\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.001425 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.001339    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-data\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.001425 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.001388    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-credential-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.001664 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.001445    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-workload-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.001664 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.001479    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-workload-certs\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.001664 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.001510    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6aa0a29-6f37-4b42-92f1-95707639c58a-istiod-ca-cert\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.001664 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.001538    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-podinfo\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.001664 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.001566    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mh27\" (UniqueName: \"kubernetes.io/projected/a6aa0a29-6f37-4b42-92f1-95707639c58a-kube-api-access-4mh27\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.102945 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.102825    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-workload-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.102945 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.102909    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-workload-certs\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.102945 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.102939    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6aa0a29-6f37-4b42-92f1-95707639c58a-istiod-ca-cert\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.103212 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.102963    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-podinfo\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.103212 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.102984    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mh27\" (UniqueName: \"kubernetes.io/projected/a6aa0a29-6f37-4b42-92f1-95707639c58a-kube-api-access-4mh27\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.103316 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.103267    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-envoy\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.103413 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.103400    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-token\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.103448 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.103434    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-data\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.103520 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.103497    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-credential-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.103915 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.103895    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-credential-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.104213 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.104195    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-workload-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.104429 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.104412    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-workload-certs\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.104950 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.104927    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6aa0a29-6f37-4b42-92f1-95707639c58a-istiod-ca-cert\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.105605 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.105578    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-data\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.107625 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.107600    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-envoy\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.107768 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.107737    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-podinfo\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.129597 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.129558    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-token\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.137444 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.137412    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mh27\" (UniqueName: \"kubernetes.io/projected/a6aa0a29-6f37-4b42-92f1-95707639c58a-kube-api-access-4mh27\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.294183 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.294138    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:23.696073 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.691207    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"]
Oct 30 19:59:23.708519 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.705764    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11080-929e15"]
Oct 30 19:59:23.712069 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.709317    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:59:23.712069 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.709430    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:59:23.712069 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.709478    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:59:23.760491 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.760460    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11080-929e15"]
Oct 30 19:59:23.763782 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.763754    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp"
Oct 30 19:59:23.840249 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.840204    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f" event={"ID":"a6aa0a29-6f37-4b42-92f1-95707639c58a","Type":"ContainerStarted","Data":"5a96b2d2ea39ce225c4f3585e4688c75559180b7d079cad0938a4fb0bffcf7d6"}
Oct 30 19:59:23.840666 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.840267    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f" event={"ID":"a6aa0a29-6f37-4b42-92f1-95707639c58a","Type":"ContainerStarted","Data":"51fd55ee6f40e1433f554eb3fc6f0ad3bc422874387be229862a633c23ff2329"}
Oct 30 19:59:23.892705 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:23.892652    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f" podStartSLOduration=1.89263618 podStartE2EDuration="1.89263618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:23.887864999 +0000 UTC m=+1718.472134622" watchObservedRunningTime="2026-10-30 19:59:23.89263618 +0000 UTC m=+1718.476905797"
Oct 30 19:59:24.294531 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:24.294498    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:24.301072 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:24.301022    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:24.848629 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:24.848573    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11080-929e15" podUID="491b27e8-f44a-455e-97c9-a35fb5f2d39e" containerName="test-curl-11080-929e15" containerID="cri-o://231d1b88746a0c37a5dd2e868b1725bb90a6c3b73ec5143c5741a45aa61ed1b6" gracePeriod=2
Oct 30 19:59:24.849151 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:24.848823    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:24.849653 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:24.849628    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 19:59:24.851706 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:59:24.851682    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11080-929e15\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="491b27e8-f44a-455e-97c9-a35fb5f2d39e" pod="opendatahub/test-curl-11080-929e15"
Oct 30 19:59:24.881205 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:59:24.881173    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11080-929e15\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="491b27e8-f44a-455e-97c9-a35fb5f2d39e" pod="opendatahub/test-curl-11080-929e15"
Oct 30 19:59:25.531669 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:59:25.531636    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11080-929e15\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="491b27e8-f44a-455e-97c9-a35fb5f2d39e" pod="opendatahub/test-curl-11080-929e15"
Oct 30 19:59:26.091088 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.091057    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"]
Oct 30 19:59:26.091824 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.091808    5408 cpu_manager.go:414] "containerMap: removing container" podUID="491b27e8-f44a-455e-97c9-a35fb5f2d39e" containerName="test-curl-11080-929e15"
Oct 30 19:59:26.091876 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.091826    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="491b27e8-f44a-455e-97c9-a35fb5f2d39e" containerName="test-curl-11080-929e15"
Oct 30 19:59:26.091978 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.091968    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="491b27e8-f44a-455e-97c9-a35fb5f2d39e" containerName="test-curl-11080-929e15"
Oct 30 19:59:26.096264 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.095896    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"
Oct 30 19:59:26.109809 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.109785    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-ae2147-serving-cert\""
Oct 30 19:59:26.111627 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.111603    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"]
Oct 30 19:59:26.160415 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.160385    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d296b553-d243-4400-8c00-3de4f7b31ff7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx\" (UID: \"d296b553-d243-4400-8c00-3de4f7b31ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"
Oct 30 19:59:26.160594 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.160579    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d296b553-d243-4400-8c00-3de4f7b31ff7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx\" (UID: \"d296b553-d243-4400-8c00-3de4f7b31ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"
Oct 30 19:59:26.160636 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.160607    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjwh8\" (UniqueName: \"kubernetes.io/projected/d296b553-d243-4400-8c00-3de4f7b31ff7-kube-api-access-jjwh8\") pod \"maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx\" (UID: \"d296b553-d243-4400-8c00-3de4f7b31ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"
Oct 30 19:59:26.160798 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.160783    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d296b553-d243-4400-8c00-3de4f7b31ff7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx\" (UID: \"d296b553-d243-4400-8c00-3de4f7b31ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"
Oct 30 19:59:26.262613 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.262581    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d296b553-d243-4400-8c00-3de4f7b31ff7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx\" (UID: \"d296b553-d243-4400-8c00-3de4f7b31ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"
Oct 30 19:59:26.262613 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.262619    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjwh8\" (UniqueName: \"kubernetes.io/projected/d296b553-d243-4400-8c00-3de4f7b31ff7-kube-api-access-jjwh8\") pod \"maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx\" (UID: \"d296b553-d243-4400-8c00-3de4f7b31ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"
Oct 30 19:59:26.262863 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.262758    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d296b553-d243-4400-8c00-3de4f7b31ff7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx\" (UID: \"d296b553-d243-4400-8c00-3de4f7b31ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"
Oct 30 19:59:26.262925 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.262864    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d296b553-d243-4400-8c00-3de4f7b31ff7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx\" (UID: \"d296b553-d243-4400-8c00-3de4f7b31ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"
Oct 30 19:59:26.263530 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.263507    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d296b553-d243-4400-8c00-3de4f7b31ff7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx\" (UID: \"d296b553-d243-4400-8c00-3de4f7b31ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"
Oct 30 19:59:26.265131 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.265106    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d296b553-d243-4400-8c00-3de4f7b31ff7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx\" (UID: \"d296b553-d243-4400-8c00-3de4f7b31ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"
Oct 30 19:59:26.265236 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.265150    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d296b553-d243-4400-8c00-3de4f7b31ff7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx\" (UID: \"d296b553-d243-4400-8c00-3de4f7b31ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"
Oct 30 19:59:26.287558 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.287534    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjwh8\" (UniqueName: \"kubernetes.io/projected/d296b553-d243-4400-8c00-3de4f7b31ff7-kube-api-access-jjwh8\") pod \"maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx\" (UID: \"d296b553-d243-4400-8c00-3de4f7b31ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"
Oct 30 19:59:26.435193 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.428340    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"
Oct 30 19:59:26.790253 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.790098    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"]
Oct 30 19:59:26.904296 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:26.904200    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx" event={"ID":"d296b553-d243-4400-8c00-3de4f7b31ff7","Type":"ContainerStarted","Data":"26a92104976cf0e9b3a78062f8d778c0443a7f764bd8db30c3aca84e746b6faa"}
Oct 30 19:59:27.530083 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:27.526276    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-929e15"
Oct 30 19:59:27.669888 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:27.664723    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/491b27e8-f44a-455e-97c9-a35fb5f2d39e-kube-api-access-w6wrf\" (UniqueName: \"kubernetes.io/projected/491b27e8-f44a-455e-97c9-a35fb5f2d39e-kube-api-access-w6wrf\") pod \"491b27e8-f44a-455e-97c9-a35fb5f2d39e\" (UID: \"491b27e8-f44a-455e-97c9-a35fb5f2d39e\") "
Oct 30 19:59:27.673091 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:27.671291    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491b27e8-f44a-455e-97c9-a35fb5f2d39e-kube-api-access-w6wrf" pod "491b27e8-f44a-455e-97c9-a35fb5f2d39e" (UID: "491b27e8-f44a-455e-97c9-a35fb5f2d39e"). InnerVolumeSpecName "kube-api-access-w6wrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:27.767301 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:27.767267    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6wrf\" (UniqueName: \"kubernetes.io/projected/491b27e8-f44a-455e-97c9-a35fb5f2d39e-kube-api-access-w6wrf\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:27.913371 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:27.913283    5408 generic.go:356] "Generic (PLEG): container finished" podID="491b27e8-f44a-455e-97c9-a35fb5f2d39e" containerID="231d1b88746a0c37a5dd2e868b1725bb90a6c3b73ec5143c5741a45aa61ed1b6" exitCode=137
Oct 30 19:59:27.913371 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:27.913333    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-929e15"
Oct 30 19:59:27.913595 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:27.913377    5408 scope.go:122] "RemoveContainer" containerID="231d1b88746a0c37a5dd2e868b1725bb90a6c3b73ec5143c5741a45aa61ed1b6"
Oct 30 19:59:27.915969 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:27.915935    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx" event={"ID":"d296b553-d243-4400-8c00-3de4f7b31ff7","Type":"ContainerStarted","Data":"4b9cd9f6fcbe1f7cf1a883f920a9b2256c6b17c6fb66f89745f15f226f0ea9b4"}
Oct 30 19:59:27.916304 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:27.916285    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"
Oct 30 19:59:27.926115 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:27.926097    5408 scope.go:122] "RemoveContainer" containerID="231d1b88746a0c37a5dd2e868b1725bb90a6c3b73ec5143c5741a45aa61ed1b6"
Oct 30 19:59:27.926404 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:59:27.926386    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231d1b88746a0c37a5dd2e868b1725bb90a6c3b73ec5143c5741a45aa61ed1b6\": container with ID starting with 231d1b88746a0c37a5dd2e868b1725bb90a6c3b73ec5143c5741a45aa61ed1b6 not found: ID does not exist" containerID="231d1b88746a0c37a5dd2e868b1725bb90a6c3b73ec5143c5741a45aa61ed1b6"
Oct 30 19:59:27.926464 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:27.926411    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231d1b88746a0c37a5dd2e868b1725bb90a6c3b73ec5143c5741a45aa61ed1b6"} err="failed to get container status \"231d1b88746a0c37a5dd2e868b1725bb90a6c3b73ec5143c5741a45aa61ed1b6\": rpc error: code = NotFound desc = could not find container \"231d1b88746a0c37a5dd2e868b1725bb90a6c3b73ec5143c5741a45aa61ed1b6\": container with ID starting with 231d1b88746a0c37a5dd2e868b1725bb90a6c3b73ec5143c5741a45aa61ed1b6 not found: ID does not exist"
Oct 30 19:59:27.945757 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:27.945705    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx" podStartSLOduration=1.3587175550000001 podStartE2EDuration="1.945677137s" totalImagesPullingTime="586.959582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:27.943556296 +0000 UTC m=+1722.527825913" watchObservedRunningTime="2026-10-30 19:59:27.945677137 +0000 UTC m=+1722.529946754"
Oct 30 19:59:29.541971 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:29.541939    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="491b27e8-f44a-455e-97c9-a35fb5f2d39e" path="/var/lib/kubelet/pods/491b27e8-f44a-455e-97c9-a35fb5f2d39e/volumes"
Oct 30 19:59:32.225401 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:32.225370    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"]
Oct 30 19:59:32.240120 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:32.240098    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 19:59:32.247589 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:32.247562    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"]
Oct 30 19:59:32.247773 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:32.247758    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c4c072-serving-cert\""
Oct 30 19:59:32.405309 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:32.405268    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-metrics-certs\") pod \"maas-api-e2e-api-b-c4c072-558d57467-2csf5\" (UID: \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 19:59:32.405885 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:32.405857    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vgvj\" (UniqueName: \"kubernetes.io/projected/54ccdd75-9a27-4c5a-9a0a-9277ec782195-kube-api-access-5vgvj\") pod \"maas-api-e2e-api-b-c4c072-558d57467-2csf5\" (UID: \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 19:59:32.411226 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:32.411191    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-maas-api-tls\") pod \"maas-api-e2e-api-b-c4c072-558d57467-2csf5\" (UID: \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 19:59:32.411750 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:32.411725    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ccdd75-9a27-4c5a-9a0a-9277ec782195-openshift-service-ca\") pod \"maas-api-e2e-api-b-c4c072-558d57467-2csf5\" (UID: \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 19:59:32.514883 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:32.514856    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vgvj\" (UniqueName: \"kubernetes.io/projected/54ccdd75-9a27-4c5a-9a0a-9277ec782195-kube-api-access-5vgvj\") pod \"maas-api-e2e-api-b-c4c072-558d57467-2csf5\" (UID: \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 19:59:32.515071 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:32.514936    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-maas-api-tls\") pod \"maas-api-e2e-api-b-c4c072-558d57467-2csf5\" (UID: \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 19:59:32.515071 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:32.515018    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ccdd75-9a27-4c5a-9a0a-9277ec782195-openshift-service-ca\") pod \"maas-api-e2e-api-b-c4c072-558d57467-2csf5\" (UID: \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 19:59:32.515155 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:59:32.515097    5408 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-serving-cert: secret "maas-api-e2e-api-b-c4c072-serving-cert" not found
Oct 30 19:59:32.515195 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:59:32.515165    5408 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-maas-api-tls podName:54ccdd75-9a27-4c5a-9a0a-9277ec782195 nodeName:}" failed. No retries permitted until 2026-10-30 19:59:33.015145421 +0000 UTC m=+1727.599415017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-maas-api-tls") pod "maas-api-e2e-api-b-c4c072-558d57467-2csf5" (UID: "54ccdd75-9a27-4c5a-9a0a-9277ec782195") : secret "maas-api-e2e-api-b-c4c072-serving-cert" not found
Oct 30 19:59:32.515384 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:32.515364    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-metrics-certs\") pod \"maas-api-e2e-api-b-c4c072-558d57467-2csf5\" (UID: \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 19:59:32.515640 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:32.515619    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ccdd75-9a27-4c5a-9a0a-9277ec782195-openshift-service-ca\") pod \"maas-api-e2e-api-b-c4c072-558d57467-2csf5\" (UID: \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 19:59:32.518262 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:32.518231    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-metrics-certs\") pod \"maas-api-e2e-api-b-c4c072-558d57467-2csf5\" (UID: \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 19:59:32.538988 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:32.538933    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vgvj\" (UniqueName: \"kubernetes.io/projected/54ccdd75-9a27-4c5a-9a0a-9277ec782195-kube-api-access-5vgvj\") pod \"maas-api-e2e-api-b-c4c072-558d57467-2csf5\" (UID: \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 19:59:33.027694 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:33.027651    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-maas-api-tls\") pod \"maas-api-e2e-api-b-c4c072-558d57467-2csf5\" (UID: \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 19:59:33.028422 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:59:33.028073    5408 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-serving-cert: secret "maas-api-e2e-api-b-c4c072-serving-cert" not found
Oct 30 19:59:33.028643 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:59:33.028630    5408 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-maas-api-tls podName:54ccdd75-9a27-4c5a-9a0a-9277ec782195 nodeName:}" failed. No retries permitted until 2026-10-30 19:59:34.028605163 +0000 UTC m=+1728.612874774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-maas-api-tls") pod "maas-api-e2e-api-b-c4c072-558d57467-2csf5" (UID: "54ccdd75-9a27-4c5a-9a0a-9277ec782195") : secret "maas-api-e2e-api-b-c4c072-serving-cert" not found
Oct 30 19:59:33.930057 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:33.930016    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"
Oct 30 19:59:34.056598 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:34.056567    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-maas-api-tls\") pod \"maas-api-e2e-api-b-c4c072-558d57467-2csf5\" (UID: \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 19:59:34.058792 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:34.058767    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-maas-api-tls\") pod \"maas-api-e2e-api-b-c4c072-558d57467-2csf5\" (UID: \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 19:59:34.075761 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:34.075726    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 19:59:34.396489 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:34.396447    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"]
Oct 30 19:59:34.407576 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:59:34.407545    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ccdd75_9a27_4c5a_9a0a_9277ec782195.slice/crio-0775b93b57966c2690cd8f53b67f4456e519611fdd67b4782030f1f76ad34c60 WatchSource:0}: Error finding container 0775b93b57966c2690cd8f53b67f4456e519611fdd67b4782030f1f76ad34c60: Status 404 returned error can't find the container with id 0775b93b57966c2690cd8f53b67f4456e519611fdd67b4782030f1f76ad34c60
Oct 30 19:59:35.125095 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:35.123469    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5" event={"ID":"54ccdd75-9a27-4c5a-9a0a-9277ec782195","Type":"ContainerStarted","Data":"0775b93b57966c2690cd8f53b67f4456e519611fdd67b4782030f1f76ad34c60"}
Oct 30 19:59:35.482690 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:35.482610    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"]
Oct 30 19:59:35.487215 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:35.487183    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 19:59:35.500068 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:35.500043    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-dd1b04-serving-cert\""
Oct 30 19:59:35.514693 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:35.513007    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"]
Oct 30 19:59:35.617097 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:35.601843    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c08ea6f-3cde-4826-b83f-1397530f484d-maas-api-tls\") pod \"maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz\" (UID: \"6c08ea6f-3cde-4826-b83f-1397530f484d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 19:59:35.617097 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:35.602020    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c08ea6f-3cde-4826-b83f-1397530f484d-openshift-service-ca\") pod \"maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz\" (UID: \"6c08ea6f-3cde-4826-b83f-1397530f484d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 19:59:35.617097 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:35.602221    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c08ea6f-3cde-4826-b83f-1397530f484d-metrics-certs\") pod \"maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz\" (UID: \"6c08ea6f-3cde-4826-b83f-1397530f484d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 19:59:35.617097 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:35.602400    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwj4n\" (UniqueName: \"kubernetes.io/projected/6c08ea6f-3cde-4826-b83f-1397530f484d-kube-api-access-rwj4n\") pod \"maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz\" (UID: \"6c08ea6f-3cde-4826-b83f-1397530f484d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 19:59:35.714054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:35.706170    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwj4n\" (UniqueName: \"kubernetes.io/projected/6c08ea6f-3cde-4826-b83f-1397530f484d-kube-api-access-rwj4n\") pod \"maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz\" (UID: \"6c08ea6f-3cde-4826-b83f-1397530f484d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 19:59:35.714054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:35.706322    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c08ea6f-3cde-4826-b83f-1397530f484d-maas-api-tls\") pod \"maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz\" (UID: \"6c08ea6f-3cde-4826-b83f-1397530f484d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 19:59:35.714054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:35.706484    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c08ea6f-3cde-4826-b83f-1397530f484d-openshift-service-ca\") pod \"maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz\" (UID: \"6c08ea6f-3cde-4826-b83f-1397530f484d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 19:59:35.714054 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:35.706538    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c08ea6f-3cde-4826-b83f-1397530f484d-metrics-certs\") pod \"maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz\" (UID: \"6c08ea6f-3cde-4826-b83f-1397530f484d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 19:59:35.729725 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:59:35.729690    5408 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-serving-cert: secret "maas-api-e2e-api-b-dd1b04-serving-cert" not found
Oct 30 19:59:35.729918 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:59:35.729787    5408 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c08ea6f-3cde-4826-b83f-1397530f484d-maas-api-tls podName:6c08ea6f-3cde-4826-b83f-1397530f484d nodeName:}" failed. No retries permitted until 2026-10-30 19:59:36.229763561 +0000 UTC m=+1730.814033172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c08ea6f-3cde-4826-b83f-1397530f484d-maas-api-tls") pod "maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz" (UID: "6c08ea6f-3cde-4826-b83f-1397530f484d") : secret "maas-api-e2e-api-b-dd1b04-serving-cert" not found
Oct 30 19:59:35.730983 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:35.730953    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c08ea6f-3cde-4826-b83f-1397530f484d-openshift-service-ca\") pod \"maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz\" (UID: \"6c08ea6f-3cde-4826-b83f-1397530f484d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 19:59:35.766880 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:35.758871    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwj4n\" (UniqueName: \"kubernetes.io/projected/6c08ea6f-3cde-4826-b83f-1397530f484d-kube-api-access-rwj4n\") pod \"maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz\" (UID: \"6c08ea6f-3cde-4826-b83f-1397530f484d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 19:59:35.766880 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:35.762168    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c08ea6f-3cde-4826-b83f-1397530f484d-metrics-certs\") pod \"maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz\" (UID: \"6c08ea6f-3cde-4826-b83f-1397530f484d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 19:59:36.155107 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:36.154391    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5" event={"ID":"54ccdd75-9a27-4c5a-9a0a-9277ec782195","Type":"ContainerStarted","Data":"ea64e23030a1ddc74df8537e77dd4c558b37f11288132e0e01af17a2bc2413d3"}
Oct 30 19:59:36.155583 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:36.155517    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 19:59:36.209055 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:36.206110    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5" podStartSLOduration=3.69224362 podStartE2EDuration="4.206091009s" totalImagesPullingTime="513.847389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:36.205654298 +0000 UTC m=+1730.789923918" watchObservedRunningTime="2026-10-30 19:59:36.206091009 +0000 UTC m=+1730.790360628"
Oct 30 19:59:36.343095 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:36.327006    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c08ea6f-3cde-4826-b83f-1397530f484d-maas-api-tls\") pod \"maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz\" (UID: \"6c08ea6f-3cde-4826-b83f-1397530f484d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 19:59:36.365052 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:36.363132    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c08ea6f-3cde-4826-b83f-1397530f484d-maas-api-tls\") pod \"maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz\" (UID: \"6c08ea6f-3cde-4826-b83f-1397530f484d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 19:59:36.418183 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:36.417409    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 19:59:36.909832 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:36.909790    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"]
Oct 30 19:59:36.926899 ip-10-0-141-19 kubenswrapper[5408]: W1030 19:59:36.926863    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c08ea6f_3cde_4826_b83f_1397530f484d.slice/crio-2ad7341c57f3da35129565830ae86f38e1199dc9ed816a96db4b5fa4d83f0279 WatchSource:0}: Error finding container 2ad7341c57f3da35129565830ae86f38e1199dc9ed816a96db4b5fa4d83f0279: Status 404 returned error can't find the container with id 2ad7341c57f3da35129565830ae86f38e1199dc9ed816a96db4b5fa4d83f0279
Oct 30 19:59:36.928864 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:36.928490    5408 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:59:37.162322 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:37.162284    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz" event={"ID":"6c08ea6f-3cde-4826-b83f-1397530f484d","Type":"ContainerStarted","Data":"2ad7341c57f3da35129565830ae86f38e1199dc9ed816a96db4b5fa4d83f0279"}
Oct 30 19:59:38.171785 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:38.171752    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz" event={"ID":"6c08ea6f-3cde-4826-b83f-1397530f484d","Type":"ContainerStarted","Data":"76eefac513db885b8ea7bda31e006ad339606d6a1f8e05f3e6db236404e3bda0"}
Oct 30 19:59:38.172225 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:38.171842    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 19:59:38.194220 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:38.194166    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz" podStartSLOduration=2.70413747 podStartE2EDuration="3.19414977s" totalImagesPullingTime="490.0123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:38.192717513 +0000 UTC m=+1732.776987130" watchObservedRunningTime="2026-10-30 19:59:38.19414977 +0000 UTC m=+1732.778419388"
Oct 30 19:59:39.748123 ip-10-0-141-19 kubenswrapper[5408]: E1030 19:59:39.743249    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 19:59:43.178096 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:43.177988    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 19:59:44.184663 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:44.184636    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 19:59:54.603175 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:54.603124    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:55.018232 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:55.018178    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p" podUID="f96b2a7d-14e7-4386-921d-b3bb1d6ec55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:55.146056 ip-10-0-141-19 kubenswrapper[5408]: I1030 19:59:55.145112    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:00:00.189953 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:00.189917    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8"]
Oct 30 20:00:00.197017 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:00.196992    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8"
Oct 30 20:00:00.205520 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:00.205496    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:00:00.217615 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:00.217580    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8"]
Oct 30 20:00:00.219180 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:00.219141    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxpmm\" (UniqueName: \"kubernetes.io/projected/8212a072-9983-47fd-9f83-08cd1a34747c-kube-api-access-mxpmm\") pod \"maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8\" (UID: \"8212a072-9983-47fd-9f83-08cd1a34747c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8"
Oct 30 20:00:00.219350 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:00.219330    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8212a072-9983-47fd-9f83-08cd1a34747c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8\" (UID: \"8212a072-9983-47fd-9f83-08cd1a34747c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8"
Oct 30 20:00:00.320703 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:00.320660    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxpmm\" (UniqueName: \"kubernetes.io/projected/8212a072-9983-47fd-9f83-08cd1a34747c-kube-api-access-mxpmm\") pod \"maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8\" (UID: \"8212a072-9983-47fd-9f83-08cd1a34747c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8"
Oct 30 20:00:00.320898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:00.320845    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8212a072-9983-47fd-9f83-08cd1a34747c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8\" (UID: \"8212a072-9983-47fd-9f83-08cd1a34747c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8"
Oct 30 20:00:00.321513 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:00.321495    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8212a072-9983-47fd-9f83-08cd1a34747c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8\" (UID: \"8212a072-9983-47fd-9f83-08cd1a34747c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8"
Oct 30 20:00:00.351777 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:00.351741    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxpmm\" (UniqueName: \"kubernetes.io/projected/8212a072-9983-47fd-9f83-08cd1a34747c-kube-api-access-mxpmm\") pod \"maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8\" (UID: \"8212a072-9983-47fd-9f83-08cd1a34747c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8"
Oct 30 20:00:00.535054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:00.522340    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8"
Oct 30 20:00:00.871617 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:00.871592    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8"]
Oct 30 20:00:00.873146 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:00:00.873112    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8212a072_9983_47fd_9f83_08cd1a34747c.slice/crio-0266a1be98f40be15697057b65331070f0290cd74f2bae1281d79bd967b760e8 WatchSource:0}: Error finding container 0266a1be98f40be15697057b65331070f0290cd74f2bae1281d79bd967b760e8: Status 404 returned error can't find the container with id 0266a1be98f40be15697057b65331070f0290cd74f2bae1281d79bd967b760e8
Oct 30 20:00:01.627219 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:01.627167    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8" event={"ID":"8212a072-9983-47fd-9f83-08cd1a34747c","Type":"ContainerStarted","Data":"0266a1be98f40be15697057b65331070f0290cd74f2bae1281d79bd967b760e8"}
Oct 30 20:00:02.232572 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:02.232524    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"]
Oct 30 20:00:02.232816 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:02.232786    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk" podUID="a2a68366-457b-4389-ac39-8c480e36abe3" containerName="maas-api" containerID="cri-o://d6d9f11ca0ec79bcd08c57ae8b38758462faaa4f7edd5fec55126ea04fb51ea0" gracePeriod=30
Oct 30 20:00:02.991051 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:02.990688    5408 generic.go:356] "Generic (PLEG): container finished" podID="a2a68366-457b-4389-ac39-8c480e36abe3" containerID="d6d9f11ca0ec79bcd08c57ae8b38758462faaa4f7edd5fec55126ea04fb51ea0" exitCode=0
Oct 30 20:00:02.991051 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:02.990965    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk" event={"ID":"a2a68366-457b-4389-ac39-8c480e36abe3","Type":"ContainerDied","Data":"d6d9f11ca0ec79bcd08c57ae8b38758462faaa4f7edd5fec55126ea04fb51ea0"}
Oct 30 20:00:03.094058 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.087871    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"
Oct 30 20:00:03.204112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.189741    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2a68366-457b-4389-ac39-8c480e36abe3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2a68366-457b-4389-ac39-8c480e36abe3-maas-api-tls\") pod \"a2a68366-457b-4389-ac39-8c480e36abe3\" (UID: \"a2a68366-457b-4389-ac39-8c480e36abe3\") "
Oct 30 20:00:03.204112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.189843    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a68366-457b-4389-ac39-8c480e36abe3-kube-api-access-c2q94\" (UniqueName: \"kubernetes.io/projected/a2a68366-457b-4389-ac39-8c480e36abe3-kube-api-access-c2q94\") pod \"a2a68366-457b-4389-ac39-8c480e36abe3\" (UID: \"a2a68366-457b-4389-ac39-8c480e36abe3\") "
Oct 30 20:00:03.204112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.189897    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2a68366-457b-4389-ac39-8c480e36abe3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a68366-457b-4389-ac39-8c480e36abe3-openshift-service-ca\") pod \"a2a68366-457b-4389-ac39-8c480e36abe3\" (UID: \"a2a68366-457b-4389-ac39-8c480e36abe3\") "
Oct 30 20:00:03.204112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.189985    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2a68366-457b-4389-ac39-8c480e36abe3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2a68366-457b-4389-ac39-8c480e36abe3-metrics-certs\") pod \"a2a68366-457b-4389-ac39-8c480e36abe3\" (UID: \"a2a68366-457b-4389-ac39-8c480e36abe3\") "
Oct 30 20:00:03.207302 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.207257    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2a68366-457b-4389-ac39-8c480e36abe3-openshift-service-ca" pod "a2a68366-457b-4389-ac39-8c480e36abe3" (UID: "a2a68366-457b-4389-ac39-8c480e36abe3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:03.263226 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.263135    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2a68366-457b-4389-ac39-8c480e36abe3-maas-api-tls" pod "a2a68366-457b-4389-ac39-8c480e36abe3" (UID: "a2a68366-457b-4389-ac39-8c480e36abe3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:03.264267 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.263550    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a68366-457b-4389-ac39-8c480e36abe3-kube-api-access-c2q94" pod "a2a68366-457b-4389-ac39-8c480e36abe3" (UID: "a2a68366-457b-4389-ac39-8c480e36abe3"). InnerVolumeSpecName "kube-api-access-c2q94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:03.287978 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.287336    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4"]
Oct 30 20:00:03.291749 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.290998    5408 cpu_manager.go:414] "containerMap: removing container" podUID="a2a68366-457b-4389-ac39-8c480e36abe3" containerName="maas-api"
Oct 30 20:00:03.291749 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.291042    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a68366-457b-4389-ac39-8c480e36abe3" containerName="maas-api"
Oct 30 20:00:03.299676 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.299651    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a68366-457b-4389-ac39-8c480e36abe3" containerName="maas-api"
Oct 30 20:00:03.302749 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.301298    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2a68366-457b-4389-ac39-8c480e36abe3-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:03.302869 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.302773    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2q94\" (UniqueName: \"kubernetes.io/projected/a2a68366-457b-4389-ac39-8c480e36abe3-kube-api-access-c2q94\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:03.302869 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.302814    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a68366-457b-4389-ac39-8c480e36abe3-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:03.320977 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.320939    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2a68366-457b-4389-ac39-8c480e36abe3-metrics-certs" pod "a2a68366-457b-4389-ac39-8c480e36abe3" (UID: "a2a68366-457b-4389-ac39-8c480e36abe3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:03.320977 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.320959    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78"]
Oct 30 20:00:03.321226 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.321157    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4"
Oct 30 20:00:03.335222 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.335198    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62"]
Oct 30 20:00:03.335605 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.335581    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78"
Oct 30 20:00:03.341735 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.341716    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62"
Oct 30 20:00:03.347931 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.347912    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr"]
Oct 30 20:00:03.354951 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.354930    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42"]
Oct 30 20:00:03.357567 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.355438    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr"
Oct 30 20:00:03.363196 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.363173    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk"]
Oct 30 20:00:03.363337 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.363293    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42"
Oct 30 20:00:03.368744 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.368725    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz"]
Oct 30 20:00:03.368833 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.368815    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk"
Oct 30 20:00:03.374219 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.374201    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk"]
Oct 30 20:00:03.376798 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.374436    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz"
Oct 30 20:00:03.381221 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.381204    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb"]
Oct 30 20:00:03.382337 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.382320    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk"
Oct 30 20:00:03.479045 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.477464    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b60350-a468-4dec-afb8-c1f67c3939e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk\" (UID: \"21b60350-a468-4dec-afb8-c1f67c3939e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk"
Oct 30 20:00:03.479045 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.477519    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9bbr\" (UniqueName: \"kubernetes.io/projected/21b60350-a468-4dec-afb8-c1f67c3939e5-kube-api-access-n9bbr\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk\" (UID: \"21b60350-a468-4dec-afb8-c1f67c3939e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk"
Oct 30 20:00:03.479045 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.477988    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2a68366-457b-4389-ac39-8c480e36abe3-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:03.499759 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.499722    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8"]
Oct 30 20:00:03.501254 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.500090    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb"
Oct 30 20:00:03.526334 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.526304    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4"]
Oct 30 20:00:03.528895 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.528775    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8"
Oct 30 20:00:03.547325 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.547298    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk"]
Oct 30 20:00:03.547509 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.547496    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42"]
Oct 30 20:00:03.547610 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.547599    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62"]
Oct 30 20:00:03.547716 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.547707    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78"]
Oct 30 20:00:03.547803 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.547794    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr"]
Oct 30 20:00:03.551647 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.551622    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk"]
Oct 30 20:00:03.556178 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.556137    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8"]
Oct 30 20:00:03.562429 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.562359    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz"]
Oct 30 20:00:03.567726 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.567704    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb"]
Oct 30 20:00:03.579476 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.579448    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgfhm\" (UniqueName: \"kubernetes.io/projected/943fe46f-f061-4c37-80e8-0958e7e145c3-kube-api-access-qgfhm\") pod \"maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62\" (UID: \"943fe46f-f061-4c37-80e8-0958e7e145c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62"
Oct 30 20:00:03.579749 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.579563    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbn7w\" (UniqueName: \"kubernetes.io/projected/29d94ec1-c6ac-42ad-8244-2df9bf785f01-kube-api-access-kbn7w\") pod \"maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr\" (UID: \"29d94ec1-c6ac-42ad-8244-2df9bf785f01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr"
Oct 30 20:00:03.579749 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.579612    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t94rv\" (UniqueName: \"kubernetes.io/projected/39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03-kube-api-access-t94rv\") pod \"maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78\" (UID: \"39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78"
Oct 30 20:00:03.579749 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.579711    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f940d6ee-3ee6-4484-8689-5ece47e6c1ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz\" (UID: \"f940d6ee-3ee6-4484-8689-5ece47e6c1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz"
Oct 30 20:00:03.579926 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.579777    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cd9l\" (UniqueName: \"kubernetes.io/projected/53f59233-5357-4386-9f05-2acfa34200dd-kube-api-access-9cd9l\") pod \"maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42\" (UID: \"53f59233-5357-4386-9f05-2acfa34200dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42"
Oct 30 20:00:03.579926 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.579913    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b60350-a468-4dec-afb8-c1f67c3939e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk\" (UID: \"21b60350-a468-4dec-afb8-c1f67c3939e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk"
Oct 30 20:00:03.579994 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.579950    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9bbr\" (UniqueName: \"kubernetes.io/projected/21b60350-a468-4dec-afb8-c1f67c3939e5-kube-api-access-n9bbr\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk\" (UID: \"21b60350-a468-4dec-afb8-c1f67c3939e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk"
Oct 30 20:00:03.580211 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.580189    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29d94ec1-c6ac-42ad-8244-2df9bf785f01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr\" (UID: \"29d94ec1-c6ac-42ad-8244-2df9bf785f01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr"
Oct 30 20:00:03.580304 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.580254    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78\" (UID: \"39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78"
Oct 30 20:00:03.580304 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.580290    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdd5f\" (UniqueName: \"kubernetes.io/projected/ac08d47c-3b2b-4825-8f49-c8283b06ad63-kube-api-access-kdd5f\") pod \"maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk\" (UID: \"ac08d47c-3b2b-4825-8f49-c8283b06ad63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk"
Oct 30 20:00:03.580415 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.580369    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac08d47c-3b2b-4825-8f49-c8283b06ad63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk\" (UID: \"ac08d47c-3b2b-4825-8f49-c8283b06ad63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk"
Oct 30 20:00:03.580415 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.580397    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a90bbcb-2fbc-4058-a5f0-3b78bde798bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4\" (UID: \"3a90bbcb-2fbc-4058-a5f0-3b78bde798bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4"
Oct 30 20:00:03.580519 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.580439    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f59233-5357-4386-9f05-2acfa34200dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42\" (UID: \"53f59233-5357-4386-9f05-2acfa34200dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42"
Oct 30 20:00:03.580519 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.580473    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcm5r\" (UniqueName: \"kubernetes.io/projected/f940d6ee-3ee6-4484-8689-5ece47e6c1ab-kube-api-access-mcm5r\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz\" (UID: \"f940d6ee-3ee6-4484-8689-5ece47e6c1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz"
Oct 30 20:00:03.580519 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.580500    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/943fe46f-f061-4c37-80e8-0958e7e145c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62\" (UID: \"943fe46f-f061-4c37-80e8-0958e7e145c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62"
Oct 30 20:00:03.580699 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.580544    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4cdk\" (UniqueName: \"kubernetes.io/projected/3a90bbcb-2fbc-4058-a5f0-3b78bde798bc-kube-api-access-v4cdk\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4\" (UID: \"3a90bbcb-2fbc-4058-a5f0-3b78bde798bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4"
Oct 30 20:00:03.580699 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.580667    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b60350-a468-4dec-afb8-c1f67c3939e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk\" (UID: \"21b60350-a468-4dec-afb8-c1f67c3939e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk"
Oct 30 20:00:03.598369 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.598331    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9bbr\" (UniqueName: \"kubernetes.io/projected/21b60350-a468-4dec-afb8-c1f67c3939e5-kube-api-access-n9bbr\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk\" (UID: \"21b60350-a468-4dec-afb8-c1f67c3939e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk"
Oct 30 20:00:03.682091 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.682058    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbn7w\" (UniqueName: \"kubernetes.io/projected/29d94ec1-c6ac-42ad-8244-2df9bf785f01-kube-api-access-kbn7w\") pod \"maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr\" (UID: \"29d94ec1-c6ac-42ad-8244-2df9bf785f01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr"
Oct 30 20:00:03.682283 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.682109    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t94rv\" (UniqueName: \"kubernetes.io/projected/39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03-kube-api-access-t94rv\") pod \"maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78\" (UID: \"39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78"
Oct 30 20:00:03.682283 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.682200    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlgk5\" (UniqueName: \"kubernetes.io/projected/b88da0c0-5e7b-42a1-9742-f88cc062c2f2-kube-api-access-zlgk5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8\" (UID: \"b88da0c0-5e7b-42a1-9742-f88cc062c2f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8"
Oct 30 20:00:03.682283 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.682235    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f940d6ee-3ee6-4484-8689-5ece47e6c1ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz\" (UID: \"f940d6ee-3ee6-4484-8689-5ece47e6c1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz"
Oct 30 20:00:03.682450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.682302    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cd9l\" (UniqueName: \"kubernetes.io/projected/53f59233-5357-4386-9f05-2acfa34200dd-kube-api-access-9cd9l\") pod \"maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42\" (UID: \"53f59233-5357-4386-9f05-2acfa34200dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42"
Oct 30 20:00:03.682824 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.682801    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88da0c0-5e7b-42a1-9742-f88cc062c2f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8\" (UID: \"b88da0c0-5e7b-42a1-9742-f88cc062c2f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8"
Oct 30 20:00:03.682937 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.682856    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29d94ec1-c6ac-42ad-8244-2df9bf785f01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr\" (UID: \"29d94ec1-c6ac-42ad-8244-2df9bf785f01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr"
Oct 30 20:00:03.682937 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.682899    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1e7396-fe2d-4cb7-a019-0f6323f3b151-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb\" (UID: \"da1e7396-fe2d-4cb7-a019-0f6323f3b151\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb"
Oct 30 20:00:03.683092 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.682946    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78\" (UID: \"39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78"
Oct 30 20:00:03.683092 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.682974    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4t6c\" (UniqueName: \"kubernetes.io/projected/da1e7396-fe2d-4cb7-a019-0f6323f3b151-kube-api-access-d4t6c\") pod \"maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb\" (UID: \"da1e7396-fe2d-4cb7-a019-0f6323f3b151\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb"
Oct 30 20:00:03.683092 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.683014    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdd5f\" (UniqueName: \"kubernetes.io/projected/ac08d47c-3b2b-4825-8f49-c8283b06ad63-kube-api-access-kdd5f\") pod \"maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk\" (UID: \"ac08d47c-3b2b-4825-8f49-c8283b06ad63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk"
Oct 30 20:00:03.683092 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.683019    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f940d6ee-3ee6-4484-8689-5ece47e6c1ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz\" (UID: \"f940d6ee-3ee6-4484-8689-5ece47e6c1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz"
Oct 30 20:00:03.683780 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.683754    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78\" (UID: \"39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78"
Oct 30 20:00:03.683885 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.683801    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac08d47c-3b2b-4825-8f49-c8283b06ad63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk\" (UID: \"ac08d47c-3b2b-4825-8f49-c8283b06ad63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk"
Oct 30 20:00:03.683885 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.683828    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a90bbcb-2fbc-4058-a5f0-3b78bde798bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4\" (UID: \"3a90bbcb-2fbc-4058-a5f0-3b78bde798bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4"
Oct 30 20:00:03.683885 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.683867    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f59233-5357-4386-9f05-2acfa34200dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42\" (UID: \"53f59233-5357-4386-9f05-2acfa34200dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42"
Oct 30 20:00:03.684086 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.683893    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcm5r\" (UniqueName: \"kubernetes.io/projected/f940d6ee-3ee6-4484-8689-5ece47e6c1ab-kube-api-access-mcm5r\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz\" (UID: \"f940d6ee-3ee6-4484-8689-5ece47e6c1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz"
Oct 30 20:00:03.684086 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.683921    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/943fe46f-f061-4c37-80e8-0958e7e145c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62\" (UID: \"943fe46f-f061-4c37-80e8-0958e7e145c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62"
Oct 30 20:00:03.684086 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.683967    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4cdk\" (UniqueName: \"kubernetes.io/projected/3a90bbcb-2fbc-4058-a5f0-3b78bde798bc-kube-api-access-v4cdk\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4\" (UID: \"3a90bbcb-2fbc-4058-a5f0-3b78bde798bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4"
Oct 30 20:00:03.684256 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.684178    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgfhm\" (UniqueName: \"kubernetes.io/projected/943fe46f-f061-4c37-80e8-0958e7e145c3-kube-api-access-qgfhm\") pod \"maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62\" (UID: \"943fe46f-f061-4c37-80e8-0958e7e145c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62"
Oct 30 20:00:03.684301 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.684277    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29d94ec1-c6ac-42ad-8244-2df9bf785f01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr\" (UID: \"29d94ec1-c6ac-42ad-8244-2df9bf785f01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr"
Oct 30 20:00:03.684531 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.684500    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac08d47c-3b2b-4825-8f49-c8283b06ad63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk\" (UID: \"ac08d47c-3b2b-4825-8f49-c8283b06ad63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk"
Oct 30 20:00:03.685021 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.684998    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/943fe46f-f061-4c37-80e8-0958e7e145c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62\" (UID: \"943fe46f-f061-4c37-80e8-0958e7e145c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62"
Oct 30 20:00:03.685277 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.685258    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f59233-5357-4386-9f05-2acfa34200dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42\" (UID: \"53f59233-5357-4386-9f05-2acfa34200dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42"
Oct 30 20:00:03.685407 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.685384    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a90bbcb-2fbc-4058-a5f0-3b78bde798bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4\" (UID: \"3a90bbcb-2fbc-4058-a5f0-3b78bde798bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4"
Oct 30 20:00:03.698079 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.698044    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdd5f\" (UniqueName: \"kubernetes.io/projected/ac08d47c-3b2b-4825-8f49-c8283b06ad63-kube-api-access-kdd5f\") pod \"maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk\" (UID: \"ac08d47c-3b2b-4825-8f49-c8283b06ad63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk"
Oct 30 20:00:03.698736 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.698698    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcm5r\" (UniqueName: \"kubernetes.io/projected/f940d6ee-3ee6-4484-8689-5ece47e6c1ab-kube-api-access-mcm5r\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz\" (UID: \"f940d6ee-3ee6-4484-8689-5ece47e6c1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz"
Oct 30 20:00:03.698836 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.698736    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbn7w\" (UniqueName: \"kubernetes.io/projected/29d94ec1-c6ac-42ad-8244-2df9bf785f01-kube-api-access-kbn7w\") pod \"maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr\" (UID: \"29d94ec1-c6ac-42ad-8244-2df9bf785f01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr"
Oct 30 20:00:03.698836 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.698719    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t94rv\" (UniqueName: \"kubernetes.io/projected/39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03-kube-api-access-t94rv\") pod \"maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78\" (UID: \"39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78"
Oct 30 20:00:03.699517 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.699498    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgfhm\" (UniqueName: \"kubernetes.io/projected/943fe46f-f061-4c37-80e8-0958e7e145c3-kube-api-access-qgfhm\") pod \"maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62\" (UID: \"943fe46f-f061-4c37-80e8-0958e7e145c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62"
Oct 30 20:00:03.700190 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.700163    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4cdk\" (UniqueName: \"kubernetes.io/projected/3a90bbcb-2fbc-4058-a5f0-3b78bde798bc-kube-api-access-v4cdk\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4\" (UID: \"3a90bbcb-2fbc-4058-a5f0-3b78bde798bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4"
Oct 30 20:00:03.701191 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.701172    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cd9l\" (UniqueName: \"kubernetes.io/projected/53f59233-5357-4386-9f05-2acfa34200dd-kube-api-access-9cd9l\") pod \"maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42\" (UID: \"53f59233-5357-4386-9f05-2acfa34200dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42"
Oct 30 20:00:03.786078 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.785944    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88da0c0-5e7b-42a1-9742-f88cc062c2f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8\" (UID: \"b88da0c0-5e7b-42a1-9742-f88cc062c2f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8"
Oct 30 20:00:03.786078 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.786012    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1e7396-fe2d-4cb7-a019-0f6323f3b151-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb\" (UID: \"da1e7396-fe2d-4cb7-a019-0f6323f3b151\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb"
Oct 30 20:00:03.786078 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.786066    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4t6c\" (UniqueName: \"kubernetes.io/projected/da1e7396-fe2d-4cb7-a019-0f6323f3b151-kube-api-access-d4t6c\") pod \"maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb\" (UID: \"da1e7396-fe2d-4cb7-a019-0f6323f3b151\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb"
Oct 30 20:00:03.786350 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.786297    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlgk5\" (UniqueName: \"kubernetes.io/projected/b88da0c0-5e7b-42a1-9742-f88cc062c2f2-kube-api-access-zlgk5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8\" (UID: \"b88da0c0-5e7b-42a1-9742-f88cc062c2f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8"
Oct 30 20:00:03.786888 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.786863    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1e7396-fe2d-4cb7-a019-0f6323f3b151-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb\" (UID: \"da1e7396-fe2d-4cb7-a019-0f6323f3b151\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb"
Oct 30 20:00:03.786888 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.786882    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88da0c0-5e7b-42a1-9742-f88cc062c2f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8\" (UID: \"b88da0c0-5e7b-42a1-9742-f88cc062c2f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8"
Oct 30 20:00:03.791233 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.791211    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr"
Oct 30 20:00:03.812290 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.812254    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42"
Oct 30 20:00:03.815631 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.815599    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlgk5\" (UniqueName: \"kubernetes.io/projected/b88da0c0-5e7b-42a1-9742-f88cc062c2f2-kube-api-access-zlgk5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8\" (UID: \"b88da0c0-5e7b-42a1-9742-f88cc062c2f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8"
Oct 30 20:00:03.815768 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.815663    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4t6c\" (UniqueName: \"kubernetes.io/projected/da1e7396-fe2d-4cb7-a019-0f6323f3b151-kube-api-access-d4t6c\") pod \"maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb\" (UID: \"da1e7396-fe2d-4cb7-a019-0f6323f3b151\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb"
Oct 30 20:00:03.828713 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.826510    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk"
Oct 30 20:00:03.842210 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.841956    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz"
Oct 30 20:00:03.854956 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.854924    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk"
Oct 30 20:00:03.866166 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.865942    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb"
Oct 30 20:00:03.880753 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.880425    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8"
Oct 30 20:00:03.957388 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.957308    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4"
Oct 30 20:00:03.967885 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.967676    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78"
Oct 30 20:00:03.981077 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:03.980826    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62"
Oct 30 20:00:04.064095 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:04.063968    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk" event={"ID":"a2a68366-457b-4389-ac39-8c480e36abe3","Type":"ContainerDied","Data":"83265c670c8dbbba94780e34b1e0fd83f785e357862028c66352c1c62355d57f"}
Oct 30 20:00:04.064095 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:04.064063    5408 scope.go:122] "RemoveContainer" containerID="d6d9f11ca0ec79bcd08c57ae8b38758462faaa4f7edd5fec55126ea04fb51ea0"
Oct 30 20:00:04.064597 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:04.064263    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"
Oct 30 20:00:04.106590 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:04.106528    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"]
Oct 30 20:00:04.114358 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:04.114279    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk"]
Oct 30 20:00:04.289837 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:04.289789    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr"]
Oct 30 20:00:04.580867 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:00:04.580836    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf940d6ee_3ee6_4484_8689_5ece47e6c1ab.slice/crio-6755e8eefcebb0f28101e183e07470befe298b5eee5587430d8578bbc41f4698 WatchSource:0}: Error finding container 6755e8eefcebb0f28101e183e07470befe298b5eee5587430d8578bbc41f4698: Status 404 returned error can't find the container with id 6755e8eefcebb0f28101e183e07470befe298b5eee5587430d8578bbc41f4698
Oct 30 20:00:04.581428 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:04.581403    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz"]
Oct 30 20:00:04.585911 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:00:04.585699    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f59233_5357_4386_9f05_2acfa34200dd.slice/crio-4cc30c15d2db7bbb3425a60164d6671dee6486114e138b17681a65b7b25d0b5d WatchSource:0}: Error finding container 4cc30c15d2db7bbb3425a60164d6671dee6486114e138b17681a65b7b25d0b5d: Status 404 returned error can't find the container with id 4cc30c15d2db7bbb3425a60164d6671dee6486114e138b17681a65b7b25d0b5d
Oct 30 20:00:04.588140 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:04.588106    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42"]
Oct 30 20:00:04.591729 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:00:04.591697    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b60350_a468_4dec_afb8_c1f67c3939e5.slice/crio-088830f89045681e68d59b87dea93b613ac3d133de85eeaebe6c6c6352355d73 WatchSource:0}: Error finding container 088830f89045681e68d59b87dea93b613ac3d133de85eeaebe6c6c6352355d73: Status 404 returned error can't find the container with id 088830f89045681e68d59b87dea93b613ac3d133de85eeaebe6c6c6352355d73
Oct 30 20:00:04.593411 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:04.593388    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk"]
Oct 30 20:00:04.708566 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:04.708540    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb"]
Oct 30 20:00:04.712816 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:04.712647    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4"]
Oct 30 20:00:04.718664 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:04.718638    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk"]
Oct 30 20:00:04.725861 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:04.724734    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8"]
Oct 30 20:00:04.736115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:04.736096    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78"]
Oct 30 20:00:04.746655 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:04.746624    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62"]
Oct 30 20:00:05.075649 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:05.075611    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62" event={"ID":"943fe46f-f061-4c37-80e8-0958e7e145c3","Type":"ContainerStarted","Data":"69239af9e2e4de96b16b746bd29f58c3704d8a16c6a6b41f51352912d3510b80"}
Oct 30 20:00:05.077550 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:05.077518    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb" event={"ID":"da1e7396-fe2d-4cb7-a019-0f6323f3b151","Type":"ContainerStarted","Data":"dba10e77bd442d7edae2c42e53474dbb120b8f64cd9c2f787d6c49e9761ef61d"}
Oct 30 20:00:05.078978 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:05.078950    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk" event={"ID":"21b60350-a468-4dec-afb8-c1f67c3939e5","Type":"ContainerStarted","Data":"088830f89045681e68d59b87dea93b613ac3d133de85eeaebe6c6c6352355d73"}
Oct 30 20:00:05.080591 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:05.080568    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8" event={"ID":"b88da0c0-5e7b-42a1-9742-f88cc062c2f2","Type":"ContainerStarted","Data":"2356bb53e6268792fd24cb7e84f673650c8ae3b61cee2eedd3d2dc92b6b259ac"}
Oct 30 20:00:05.083337 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:05.083312    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78" event={"ID":"39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03","Type":"ContainerStarted","Data":"a751eb30f5ba2659f58f26ae6d1bfb02bb33025d2552e80e881d3f521998c66e"}
Oct 30 20:00:05.084570 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:05.084544    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz" event={"ID":"f940d6ee-3ee6-4484-8689-5ece47e6c1ab","Type":"ContainerStarted","Data":"6755e8eefcebb0f28101e183e07470befe298b5eee5587430d8578bbc41f4698"}
Oct 30 20:00:05.086474 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:05.086451    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4" event={"ID":"3a90bbcb-2fbc-4058-a5f0-3b78bde798bc","Type":"ContainerStarted","Data":"2481b1ce898fd0cfa198872cd5a9afcf94e134fbf95beb77a4239bc96db47ece"}
Oct 30 20:00:05.087873 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:05.087849    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk" event={"ID":"ac08d47c-3b2b-4825-8f49-c8283b06ad63","Type":"ContainerStarted","Data":"510e4f41d4a6021b6464e77167742e48672e045c23c693160323aad16049afb5"}
Oct 30 20:00:05.089230 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:05.089209    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42" event={"ID":"53f59233-5357-4386-9f05-2acfa34200dd","Type":"ContainerStarted","Data":"4cc30c15d2db7bbb3425a60164d6671dee6486114e138b17681a65b7b25d0b5d"}
Oct 30 20:00:05.090597 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:05.090573    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr" event={"ID":"29d94ec1-c6ac-42ad-8244-2df9bf785f01","Type":"ContainerStarted","Data":"a7c9e01eb09e0866398e3732da0700a0d3d2af131b3f0d39af38a41c544af007"}
Oct 30 20:00:05.542723 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:05.542246    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2a68366-457b-4389-ac39-8c480e36abe3" path="/var/lib/kubelet/pods/a2a68366-457b-4389-ac39-8c480e36abe3/volumes"
Oct 30 20:00:05.596974 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:05.595807    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-vx2sk" podUID="a2a68366-457b-4389-ac39-8c480e36abe3" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.97:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:00:07.272757 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:07.272662    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42" event={"ID":"53f59233-5357-4386-9f05-2acfa34200dd","Type":"ContainerStarted","Data":"d80958686c4934398a66cec1c9896bfadf8a5e861cb5c15c7a2eb401a5e34b2f"}
Oct 30 20:00:07.275799 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:07.275769    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8" event={"ID":"8212a072-9983-47fd-9f83-08cd1a34747c","Type":"ContainerStarted","Data":"fa97e8b2993c528e9f6270f7d3a3eb31dbfdc5877e755038318ce5640388526e"}
Oct 30 20:00:07.327363 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:07.327307    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8" podStartSLOduration=2.259609195 podStartE2EDuration="7.327288444s" totalImagesPullingTime="5.067679249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:07.325888632 +0000 UTC m=+1761.910158252" watchObservedRunningTime="2026-10-30 20:00:07.327288444 +0000 UTC m=+1761.911558065"
Oct 30 20:00:07.328147 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:07.328101    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42" podStartSLOduration=5.9733093 podStartE2EDuration="7.328090541s" totalImagesPullingTime="1.354781241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:07.300727442 +0000 UTC m=+1761.884997059" watchObservedRunningTime="2026-10-30 20:00:07.328090541 +0000 UTC m=+1761.912360156"
Oct 30 20:00:08.285812 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:08.285731    5408 generic.go:356] "Generic (PLEG): container finished" podID="53f59233-5357-4386-9f05-2acfa34200dd" containerID="d80958686c4934398a66cec1c9896bfadf8a5e861cb5c15c7a2eb401a5e34b2f" exitCode=60
Oct 30 20:00:08.286263 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:08.285814    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42" event={"ID":"53f59233-5357-4386-9f05-2acfa34200dd","Type":"ContainerDied","Data":"d80958686c4934398a66cec1c9896bfadf8a5e861cb5c15c7a2eb401a5e34b2f"}
Oct 30 20:00:08.286263 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:08.286172    5408 scope.go:122] "RemoveContainer" containerID="d80958686c4934398a66cec1c9896bfadf8a5e861cb5c15c7a2eb401a5e34b2f"
Oct 30 20:00:08.287720 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:08.287692    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr" event={"ID":"29d94ec1-c6ac-42ad-8244-2df9bf785f01","Type":"ContainerStarted","Data":"5c70c2f6f2a998a87a7dc6d60a679337a940a0fc87807e198ab8b6125609fdb9"}
Oct 30 20:00:08.290537 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:08.289917    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62" event={"ID":"943fe46f-f061-4c37-80e8-0958e7e145c3","Type":"ContainerStarted","Data":"1c2f75b335925c5456e61e1fc0358b48598baa3791b01d664cc2d38732d2307b"}
Oct 30 20:00:08.293024 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:08.293001    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78" event={"ID":"39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03","Type":"ContainerStarted","Data":"cf5925919dbc093072e8f9a1cd6338c09fd98562f72d0fd9cd1a1695f0e1f115"}
Oct 30 20:00:08.294881 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:08.294861    5408 generic.go:356] "Generic (PLEG): container finished" podID="8212a072-9983-47fd-9f83-08cd1a34747c" containerID="fa97e8b2993c528e9f6270f7d3a3eb31dbfdc5877e755038318ce5640388526e" exitCode=60
Oct 30 20:00:08.294981 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:08.294923    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8" event={"ID":"8212a072-9983-47fd-9f83-08cd1a34747c","Type":"ContainerDied","Data":"fa97e8b2993c528e9f6270f7d3a3eb31dbfdc5877e755038318ce5640388526e"}
Oct 30 20:00:08.295156 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:08.295140    5408 scope.go:122] "RemoveContainer" containerID="fa97e8b2993c528e9f6270f7d3a3eb31dbfdc5877e755038318ce5640388526e"
Oct 30 20:00:08.397360 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:08.394897    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62" podStartSLOduration=5.419558124 podStartE2EDuration="8.394882501s" totalImagesPullingTime="2.975324377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:08.369927922 +0000 UTC m=+1762.954197552" watchObservedRunningTime="2026-10-30 20:00:08.394882501 +0000 UTC m=+1762.979152117"
Oct 30 20:00:08.458861 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:08.458810    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr" podStartSLOduration=5.483075338 podStartE2EDuration="8.458788892s" totalImagesPullingTime="2.975713554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:08.439076103 +0000 UTC m=+1763.023345721" watchObservedRunningTime="2026-10-30 20:00:08.458788892 +0000 UTC m=+1763.043058509"
Oct 30 20:00:08.492700 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:08.492657    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78" podStartSLOduration=5.4628962 podStartE2EDuration="8.492643953s" totalImagesPullingTime="3.029747753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:08.49256461 +0000 UTC m=+1763.076834228" watchObservedRunningTime="2026-10-30 20:00:08.492643953 +0000 UTC m=+1763.076913570"
Oct 30 20:00:09.361054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:09.353762    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42" event={"ID":"53f59233-5357-4386-9f05-2acfa34200dd","Type":"ContainerStarted","Data":"d6c88fc9a037e06aff2f2b26646aecbf72775ff388f6385d576857ff6f202f47"}
Oct 30 20:00:09.361054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:09.357454    5408 generic.go:356] "Generic (PLEG): container finished" podID="29d94ec1-c6ac-42ad-8244-2df9bf785f01" containerID="5c70c2f6f2a998a87a7dc6d60a679337a940a0fc87807e198ab8b6125609fdb9" exitCode=60
Oct 30 20:00:09.361054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:09.357550    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr" event={"ID":"29d94ec1-c6ac-42ad-8244-2df9bf785f01","Type":"ContainerDied","Data":"5c70c2f6f2a998a87a7dc6d60a679337a940a0fc87807e198ab8b6125609fdb9"}
Oct 30 20:00:09.361054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:09.357980    5408 scope.go:122] "RemoveContainer" containerID="5c70c2f6f2a998a87a7dc6d60a679337a940a0fc87807e198ab8b6125609fdb9"
Oct 30 20:00:09.362954 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:09.362924    5408 generic.go:356] "Generic (PLEG): container finished" podID="943fe46f-f061-4c37-80e8-0958e7e145c3" containerID="1c2f75b335925c5456e61e1fc0358b48598baa3791b01d664cc2d38732d2307b" exitCode=60
Oct 30 20:00:09.363235 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:09.363213    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62" event={"ID":"943fe46f-f061-4c37-80e8-0958e7e145c3","Type":"ContainerDied","Data":"1c2f75b335925c5456e61e1fc0358b48598baa3791b01d664cc2d38732d2307b"}
Oct 30 20:00:09.363781 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:09.363762    5408 scope.go:122] "RemoveContainer" containerID="1c2f75b335925c5456e61e1fc0358b48598baa3791b01d664cc2d38732d2307b"
Oct 30 20:00:09.454839 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:09.454782    5408 generic.go:356] "Generic (PLEG): container finished" podID="da1e7396-fe2d-4cb7-a019-0f6323f3b151" containerID="60e4874f4fbc6f97fad7dc2944c70f06f7741c69e0bd006bfe4efcf90a4a2c56" exitCode=60
Oct 30 20:00:09.454983 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:09.454963    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb" event={"ID":"da1e7396-fe2d-4cb7-a019-0f6323f3b151","Type":"ContainerDied","Data":"60e4874f4fbc6f97fad7dc2944c70f06f7741c69e0bd006bfe4efcf90a4a2c56"}
Oct 30 20:00:09.455341 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:09.455323    5408 scope.go:122] "RemoveContainer" containerID="60e4874f4fbc6f97fad7dc2944c70f06f7741c69e0bd006bfe4efcf90a4a2c56"
Oct 30 20:00:09.457137 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:09.457114    5408 generic.go:356] "Generic (PLEG): container finished" podID="39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03" containerID="cf5925919dbc093072e8f9a1cd6338c09fd98562f72d0fd9cd1a1695f0e1f115" exitCode=60
Oct 30 20:00:09.457267 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:09.457202    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78" event={"ID":"39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03","Type":"ContainerDied","Data":"cf5925919dbc093072e8f9a1cd6338c09fd98562f72d0fd9cd1a1695f0e1f115"}
Oct 30 20:00:09.457544 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:09.457524    5408 scope.go:122] "RemoveContainer" containerID="cf5925919dbc093072e8f9a1cd6338c09fd98562f72d0fd9cd1a1695f0e1f115"
Oct 30 20:00:09.459937 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:09.459916    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8" event={"ID":"8212a072-9983-47fd-9f83-08cd1a34747c","Type":"ContainerStarted","Data":"05a7a29badcc2eec02d7caac041b44845eeb851c72a2a5914e22483562e73f4c"}
Oct 30 20:00:09.684724 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:09.684685    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:00:09.799294 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:09.799255    5408 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f59233_5357_4386_9f05_2acfa34200dd.slice/crio-conmon-d6c88fc9a037e06aff2f2b26646aecbf72775ff388f6385d576857ff6f202f47.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f59233_5357_4386_9f05_2acfa34200dd.slice/crio-d6c88fc9a037e06aff2f2b26646aecbf72775ff388f6385d576857ff6f202f47.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:00:10.496084 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:10.494665    5408 generic.go:356] "Generic (PLEG): container finished" podID="8212a072-9983-47fd-9f83-08cd1a34747c" containerID="05a7a29badcc2eec02d7caac041b44845eeb851c72a2a5914e22483562e73f4c" exitCode=60
Oct 30 20:00:10.496084 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:10.494865    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8" event={"ID":"8212a072-9983-47fd-9f83-08cd1a34747c","Type":"ContainerDied","Data":"05a7a29badcc2eec02d7caac041b44845eeb851c72a2a5914e22483562e73f4c"}
Oct 30 20:00:10.496084 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:10.494988    5408 scope.go:122] "RemoveContainer" containerID="fa97e8b2993c528e9f6270f7d3a3eb31dbfdc5877e755038318ce5640388526e"
Oct 30 20:00:10.496084 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:10.495616    5408 scope.go:122] "RemoveContainer" containerID="05a7a29badcc2eec02d7caac041b44845eeb851c72a2a5914e22483562e73f4c"
Oct 30 20:00:10.497489 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:10.496102    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8_odh-ai-gateway-infra(8212a072-9983-47fd-9f83-08cd1a34747c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8" podUID="8212a072-9983-47fd-9f83-08cd1a34747c"
Oct 30 20:00:10.508577 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:10.508546    5408 generic.go:356] "Generic (PLEG): container finished" podID="f940d6ee-3ee6-4484-8689-5ece47e6c1ab" containerID="94909c460ff5b3662fb7b406ff8744a68f3db77fd7831aba3f147f27df84a245" exitCode=60
Oct 30 20:00:10.509384 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:10.508890    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz" event={"ID":"f940d6ee-3ee6-4484-8689-5ece47e6c1ab","Type":"ContainerDied","Data":"94909c460ff5b3662fb7b406ff8744a68f3db77fd7831aba3f147f27df84a245"}
Oct 30 20:00:10.509384 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:10.509199    5408 scope.go:122] "RemoveContainer" containerID="94909c460ff5b3662fb7b406ff8744a68f3db77fd7831aba3f147f27df84a245"
Oct 30 20:00:10.512891 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:10.512867    5408 generic.go:356] "Generic (PLEG): container finished" podID="53f59233-5357-4386-9f05-2acfa34200dd" containerID="d6c88fc9a037e06aff2f2b26646aecbf72775ff388f6385d576857ff6f202f47" exitCode=60
Oct 30 20:00:10.513159 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:10.513123    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42" event={"ID":"53f59233-5357-4386-9f05-2acfa34200dd","Type":"ContainerDied","Data":"d6c88fc9a037e06aff2f2b26646aecbf72775ff388f6385d576857ff6f202f47"}
Oct 30 20:00:10.513688 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:10.513668    5408 scope.go:122] "RemoveContainer" containerID="d6c88fc9a037e06aff2f2b26646aecbf72775ff388f6385d576857ff6f202f47"
Oct 30 20:00:10.514207 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:10.514174    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42_odh-ai-gateway-infra(53f59233-5357-4386-9f05-2acfa34200dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42" podUID="53f59233-5357-4386-9f05-2acfa34200dd"
Oct 30 20:00:10.517793 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:10.517770    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62" event={"ID":"943fe46f-f061-4c37-80e8-0958e7e145c3","Type":"ContainerStarted","Data":"01cdd02cd66e6487c52af0662fe764ff3be7f729cfaa1725c74254f29cf5e14c"}
Oct 30 20:00:10.534532 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:10.532735    5408 scope.go:122] "RemoveContainer" containerID="d80958686c4934398a66cec1c9896bfadf8a5e861cb5c15c7a2eb401a5e34b2f"
Oct 30 20:00:10.535355 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:10.534593    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb" event={"ID":"da1e7396-fe2d-4cb7-a019-0f6323f3b151","Type":"ContainerStarted","Data":"3b2c357044a9a0a2c250a7ee7caa239d6a867d441536b5aa533489158067d77f"}
Oct 30 20:00:10.671333 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:10.671288    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb" podStartSLOduration=7.624622348 podStartE2EDuration="10.671274015s" totalImagesPullingTime="3.046651667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:10.667472254 +0000 UTC m=+1765.251741872" watchObservedRunningTime="2026-10-30 20:00:10.671274015 +0000 UTC m=+1765.255543632"
Oct 30 20:00:11.543590 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.543552    5408 generic.go:356] "Generic (PLEG): container finished" podID="39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03" containerID="506bc9c98f62f9e6a2b6aa5bcd9592b4f8e9a58b02ed5bc272d2b0949112b9f7" exitCode=60
Oct 30 20:00:11.544002 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.543635    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78" event={"ID":"39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03","Type":"ContainerDied","Data":"506bc9c98f62f9e6a2b6aa5bcd9592b4f8e9a58b02ed5bc272d2b0949112b9f7"}
Oct 30 20:00:11.544002 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.543701    5408 scope.go:122] "RemoveContainer" containerID="cf5925919dbc093072e8f9a1cd6338c09fd98562f72d0fd9cd1a1695f0e1f115"
Oct 30 20:00:11.544002 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.543948    5408 scope.go:122] "RemoveContainer" containerID="506bc9c98f62f9e6a2b6aa5bcd9592b4f8e9a58b02ed5bc272d2b0949112b9f7"
Oct 30 20:00:11.544435 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:11.544409    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78_odh-ai-gateway-infra(39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78" podUID="39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03"
Oct 30 20:00:11.546821 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.546800    5408 scope.go:122] "RemoveContainer" containerID="05a7a29badcc2eec02d7caac041b44845eeb851c72a2a5914e22483562e73f4c"
Oct 30 20:00:11.547239 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:11.547218    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8_odh-ai-gateway-infra(8212a072-9983-47fd-9f83-08cd1a34747c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8" podUID="8212a072-9983-47fd-9f83-08cd1a34747c"
Oct 30 20:00:11.549261 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.549232    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz" event={"ID":"f940d6ee-3ee6-4484-8689-5ece47e6c1ab","Type":"ContainerStarted","Data":"94a6354dd54e9a2c74866ba37fb12d8c9ab15adb578b3c21c645b05ba3a9e5db"}
Oct 30 20:00:11.550845 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.550821    5408 generic.go:356] "Generic (PLEG): container finished" podID="3a90bbcb-2fbc-4058-a5f0-3b78bde798bc" containerID="081b317d29d70a8aa664d67c224a0cf14f73b81b3abb7a630bbf28a4120d95e0" exitCode=60
Oct 30 20:00:11.550949 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.550897    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4" event={"ID":"3a90bbcb-2fbc-4058-a5f0-3b78bde798bc","Type":"ContainerDied","Data":"081b317d29d70a8aa664d67c224a0cf14f73b81b3abb7a630bbf28a4120d95e0"}
Oct 30 20:00:11.551193 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.551173    5408 scope.go:122] "RemoveContainer" containerID="081b317d29d70a8aa664d67c224a0cf14f73b81b3abb7a630bbf28a4120d95e0"
Oct 30 20:00:11.552774 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.552751    5408 generic.go:356] "Generic (PLEG): container finished" podID="ac08d47c-3b2b-4825-8f49-c8283b06ad63" containerID="59bc945fc0e81fccc539180030288440c9add866256d481b8598f7b6a6fa9595" exitCode=60
Oct 30 20:00:11.552883 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.552784    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk" event={"ID":"ac08d47c-3b2b-4825-8f49-c8283b06ad63","Type":"ContainerDied","Data":"59bc945fc0e81fccc539180030288440c9add866256d481b8598f7b6a6fa9595"}
Oct 30 20:00:11.553110 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.553087    5408 scope.go:122] "RemoveContainer" containerID="59bc945fc0e81fccc539180030288440c9add866256d481b8598f7b6a6fa9595"
Oct 30 20:00:11.554903 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.554885    5408 scope.go:122] "RemoveContainer" containerID="d6c88fc9a037e06aff2f2b26646aecbf72775ff388f6385d576857ff6f202f47"
Oct 30 20:00:11.555154 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:11.555132    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42_odh-ai-gateway-infra(53f59233-5357-4386-9f05-2acfa34200dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42" podUID="53f59233-5357-4386-9f05-2acfa34200dd"
Oct 30 20:00:11.556574 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.556553    5408 generic.go:356] "Generic (PLEG): container finished" podID="29d94ec1-c6ac-42ad-8244-2df9bf785f01" containerID="0e2e4ac817adb750b38561d2debbd68bd07766df9c7c6c8b60d7a557c8b50c4e" exitCode=60
Oct 30 20:00:11.556666 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.556630    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr" event={"ID":"29d94ec1-c6ac-42ad-8244-2df9bf785f01","Type":"ContainerDied","Data":"0e2e4ac817adb750b38561d2debbd68bd07766df9c7c6c8b60d7a557c8b50c4e"}
Oct 30 20:00:11.556879 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.556860    5408 scope.go:122] "RemoveContainer" containerID="0e2e4ac817adb750b38561d2debbd68bd07766df9c7c6c8b60d7a557c8b50c4e"
Oct 30 20:00:11.557120 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:11.557098    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr_odh-ai-gateway-infra(29d94ec1-c6ac-42ad-8244-2df9bf785f01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr" podUID="29d94ec1-c6ac-42ad-8244-2df9bf785f01"
Oct 30 20:00:11.558594 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.558567    5408 generic.go:356] "Generic (PLEG): container finished" podID="943fe46f-f061-4c37-80e8-0958e7e145c3" containerID="01cdd02cd66e6487c52af0662fe764ff3be7f729cfaa1725c74254f29cf5e14c" exitCode=60
Oct 30 20:00:11.558672 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.558644    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62" event={"ID":"943fe46f-f061-4c37-80e8-0958e7e145c3","Type":"ContainerDied","Data":"01cdd02cd66e6487c52af0662fe764ff3be7f729cfaa1725c74254f29cf5e14c"}
Oct 30 20:00:11.558819 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.558804    5408 scope.go:122] "RemoveContainer" containerID="01cdd02cd66e6487c52af0662fe764ff3be7f729cfaa1725c74254f29cf5e14c"
Oct 30 20:00:11.559040 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:11.559007    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62_odh-ai-gateway-infra(943fe46f-f061-4c37-80e8-0958e7e145c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62" podUID="943fe46f-f061-4c37-80e8-0958e7e145c3"
Oct 30 20:00:11.560181 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.560163    5408 scope.go:122] "RemoveContainer" containerID="5c70c2f6f2a998a87a7dc6d60a679337a940a0fc87807e198ab8b6125609fdb9"
Oct 30 20:00:11.562343 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.562218    5408 generic.go:356] "Generic (PLEG): container finished" podID="da1e7396-fe2d-4cb7-a019-0f6323f3b151" containerID="3b2c357044a9a0a2c250a7ee7caa239d6a867d441536b5aa533489158067d77f" exitCode=60
Oct 30 20:00:11.562343 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.562297    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb" event={"ID":"da1e7396-fe2d-4cb7-a019-0f6323f3b151","Type":"ContainerDied","Data":"3b2c357044a9a0a2c250a7ee7caa239d6a867d441536b5aa533489158067d77f"}
Oct 30 20:00:11.562607 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.562591    5408 scope.go:122] "RemoveContainer" containerID="3b2c357044a9a0a2c250a7ee7caa239d6a867d441536b5aa533489158067d77f"
Oct 30 20:00:11.563382 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:11.563359    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb_odh-ai-gateway-infra(da1e7396-fe2d-4cb7-a019-0f6323f3b151)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb" podUID="da1e7396-fe2d-4cb7-a019-0f6323f3b151"
Oct 30 20:00:11.564093 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.564071    5408 generic.go:356] "Generic (PLEG): container finished" podID="21b60350-a468-4dec-afb8-c1f67c3939e5" containerID="346f70ddf4304271c895db1df7cfe6f309d9b3d45e621b71abcc0ca8ca6ec260" exitCode=60
Oct 30 20:00:11.564189 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.564135    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk" event={"ID":"21b60350-a468-4dec-afb8-c1f67c3939e5","Type":"ContainerDied","Data":"346f70ddf4304271c895db1df7cfe6f309d9b3d45e621b71abcc0ca8ca6ec260"}
Oct 30 20:00:11.564306 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.564292    5408 scope.go:122] "RemoveContainer" containerID="346f70ddf4304271c895db1df7cfe6f309d9b3d45e621b71abcc0ca8ca6ec260"
Oct 30 20:00:11.581147 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.581122    5408 scope.go:122] "RemoveContainer" containerID="1c2f75b335925c5456e61e1fc0358b48598baa3791b01d664cc2d38732d2307b"
Oct 30 20:00:11.790585 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.773791    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz" podStartSLOduration=7.269146683 podStartE2EDuration="11.773773832s" totalImagesPullingTime="4.504627149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:11.637808799 +0000 UTC m=+1766.222078419" watchObservedRunningTime="2026-10-30 20:00:11.773773832 +0000 UTC m=+1766.358043466"
Oct 30 20:00:11.831393 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:11.824536    5408 scope.go:122] "RemoveContainer" containerID="60e4874f4fbc6f97fad7dc2944c70f06f7741c69e0bd006bfe4efcf90a4a2c56"
Oct 30 20:00:12.574769 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:12.574736    5408 scope.go:122] "RemoveContainer" containerID="01cdd02cd66e6487c52af0662fe764ff3be7f729cfaa1725c74254f29cf5e14c"
Oct 30 20:00:12.575267 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:12.574996    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62_odh-ai-gateway-infra(943fe46f-f061-4c37-80e8-0958e7e145c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62" podUID="943fe46f-f061-4c37-80e8-0958e7e145c3"
Oct 30 20:00:12.576318 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:12.576299    5408 scope.go:122] "RemoveContainer" containerID="3b2c357044a9a0a2c250a7ee7caa239d6a867d441536b5aa533489158067d77f"
Oct 30 20:00:12.576534 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:12.576513    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb_odh-ai-gateway-infra(da1e7396-fe2d-4cb7-a019-0f6323f3b151)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb" podUID="da1e7396-fe2d-4cb7-a019-0f6323f3b151"
Oct 30 20:00:12.577941 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:12.577922    5408 scope.go:122] "RemoveContainer" containerID="506bc9c98f62f9e6a2b6aa5bcd9592b4f8e9a58b02ed5bc272d2b0949112b9f7"
Oct 30 20:00:12.578171 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:12.578149    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78_odh-ai-gateway-infra(39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78" podUID="39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03"
Oct 30 20:00:12.579549 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:12.579527    5408 generic.go:356] "Generic (PLEG): container finished" podID="f940d6ee-3ee6-4484-8689-5ece47e6c1ab" containerID="94a6354dd54e9a2c74866ba37fb12d8c9ab15adb578b3c21c645b05ba3a9e5db" exitCode=60
Oct 30 20:00:12.579635 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:12.579604    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz" event={"ID":"f940d6ee-3ee6-4484-8689-5ece47e6c1ab","Type":"ContainerDied","Data":"94a6354dd54e9a2c74866ba37fb12d8c9ab15adb578b3c21c645b05ba3a9e5db"}
Oct 30 20:00:12.579692 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:12.579656    5408 scope.go:122] "RemoveContainer" containerID="94909c460ff5b3662fb7b406ff8744a68f3db77fd7831aba3f147f27df84a245"
Oct 30 20:00:12.579792 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:12.579773    5408 scope.go:122] "RemoveContainer" containerID="94a6354dd54e9a2c74866ba37fb12d8c9ab15adb578b3c21c645b05ba3a9e5db"
Oct 30 20:00:12.580040 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:12.580000    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz_odh-ai-gateway-infra(f940d6ee-3ee6-4484-8689-5ece47e6c1ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz" podUID="f940d6ee-3ee6-4484-8689-5ece47e6c1ab"
Oct 30 20:00:12.581573 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:12.581548    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4" event={"ID":"3a90bbcb-2fbc-4058-a5f0-3b78bde798bc","Type":"ContainerStarted","Data":"d3f4b33c95b045068e438dc4f0ce3189861db2f54dd814579bffb66c069593f1"}
Oct 30 20:00:12.583527 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:12.583504    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk" event={"ID":"ac08d47c-3b2b-4825-8f49-c8283b06ad63","Type":"ContainerStarted","Data":"f2a67c60f85175176b0ce7785f86c91e1845b609517a5cc4fcbf26c26efb9421"}
Oct 30 20:00:12.585504 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:12.585489    5408 scope.go:122] "RemoveContainer" containerID="0e2e4ac817adb750b38561d2debbd68bd07766df9c7c6c8b60d7a557c8b50c4e"
Oct 30 20:00:12.585719 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:12.585694    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr_odh-ai-gateway-infra(29d94ec1-c6ac-42ad-8244-2df9bf785f01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr" podUID="29d94ec1-c6ac-42ad-8244-2df9bf785f01"
Oct 30 20:00:12.689386 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:12.689283    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk" podStartSLOduration=7.690132998 podStartE2EDuration="12.689270808s" totalImagesPullingTime="4.99913781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:12.687142215 +0000 UTC m=+1767.271411833" watchObservedRunningTime="2026-10-30 20:00:12.689270808 +0000 UTC m=+1767.273540425"
Oct 30 20:00:12.780409 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:12.780358    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4" podStartSLOduration=7.546292905 podStartE2EDuration="12.780342788s" totalImagesPullingTime="5.234049883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:12.757543549 +0000 UTC m=+1767.341813165" watchObservedRunningTime="2026-10-30 20:00:12.780342788 +0000 UTC m=+1767.364612404"
Oct 30 20:00:13.596921 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:13.596891    5408 generic.go:356] "Generic (PLEG): container finished" podID="ac08d47c-3b2b-4825-8f49-c8283b06ad63" containerID="f2a67c60f85175176b0ce7785f86c91e1845b609517a5cc4fcbf26c26efb9421" exitCode=60
Oct 30 20:00:13.597404 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:13.596952    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk" event={"ID":"ac08d47c-3b2b-4825-8f49-c8283b06ad63","Type":"ContainerDied","Data":"f2a67c60f85175176b0ce7785f86c91e1845b609517a5cc4fcbf26c26efb9421"}
Oct 30 20:00:13.597404 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:13.597010    5408 scope.go:122] "RemoveContainer" containerID="59bc945fc0e81fccc539180030288440c9add866256d481b8598f7b6a6fa9595"
Oct 30 20:00:13.597404 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:13.597377    5408 scope.go:122] "RemoveContainer" containerID="f2a67c60f85175176b0ce7785f86c91e1845b609517a5cc4fcbf26c26efb9421"
Oct 30 20:00:13.597802 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:13.597774    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk_odh-ai-gateway-infra(ac08d47c-3b2b-4825-8f49-c8283b06ad63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk" podUID="ac08d47c-3b2b-4825-8f49-c8283b06ad63"
Oct 30 20:00:13.599308 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:13.599283    5408 generic.go:356] "Generic (PLEG): container finished" podID="21b60350-a468-4dec-afb8-c1f67c3939e5" containerID="06f917e2f8441f24e923a47ffcf177b6b7b1989d5b236d59b8a66fa829a4b646" exitCode=60
Oct 30 20:00:13.599497 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:13.599470    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk" event={"ID":"21b60350-a468-4dec-afb8-c1f67c3939e5","Type":"ContainerDied","Data":"06f917e2f8441f24e923a47ffcf177b6b7b1989d5b236d59b8a66fa829a4b646"}
Oct 30 20:00:13.599785 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:13.599769    5408 scope.go:122] "RemoveContainer" containerID="06f917e2f8441f24e923a47ffcf177b6b7b1989d5b236d59b8a66fa829a4b646"
Oct 30 20:00:13.600138 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:13.600113    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk_odh-ai-gateway-infra(21b60350-a468-4dec-afb8-c1f67c3939e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk" podUID="21b60350-a468-4dec-afb8-c1f67c3939e5"
Oct 30 20:00:13.602134 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:13.602114    5408 scope.go:122] "RemoveContainer" containerID="94a6354dd54e9a2c74866ba37fb12d8c9ab15adb578b3c21c645b05ba3a9e5db"
Oct 30 20:00:13.602498 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:13.602473    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz_odh-ai-gateway-infra(f940d6ee-3ee6-4484-8689-5ece47e6c1ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz" podUID="f940d6ee-3ee6-4484-8689-5ece47e6c1ab"
Oct 30 20:00:13.604260 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:13.604236    5408 generic.go:356] "Generic (PLEG): container finished" podID="3a90bbcb-2fbc-4058-a5f0-3b78bde798bc" containerID="d3f4b33c95b045068e438dc4f0ce3189861db2f54dd814579bffb66c069593f1" exitCode=60
Oct 30 20:00:13.604348 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:13.604269    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4" event={"ID":"3a90bbcb-2fbc-4058-a5f0-3b78bde798bc","Type":"ContainerDied","Data":"d3f4b33c95b045068e438dc4f0ce3189861db2f54dd814579bffb66c069593f1"}
Oct 30 20:00:13.604644 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:13.604624    5408 scope.go:122] "RemoveContainer" containerID="d3f4b33c95b045068e438dc4f0ce3189861db2f54dd814579bffb66c069593f1"
Oct 30 20:00:13.605022 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:13.604997    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4_odh-ai-gateway-infra(3a90bbcb-2fbc-4058-a5f0-3b78bde798bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4" podUID="3a90bbcb-2fbc-4058-a5f0-3b78bde798bc"
Oct 30 20:00:13.613222 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:13.613204    5408 scope.go:122] "RemoveContainer" containerID="346f70ddf4304271c895db1df7cfe6f309d9b3d45e621b71abcc0ca8ca6ec260"
Oct 30 20:00:13.636318 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:13.636301    5408 scope.go:122] "RemoveContainer" containerID="081b317d29d70a8aa664d67c224a0cf14f73b81b3abb7a630bbf28a4120d95e0"
Oct 30 20:00:14.614755 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:14.614730    5408 scope.go:122] "RemoveContainer" containerID="d3f4b33c95b045068e438dc4f0ce3189861db2f54dd814579bffb66c069593f1"
Oct 30 20:00:14.615287 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:14.615106    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4_odh-ai-gateway-infra(3a90bbcb-2fbc-4058-a5f0-3b78bde798bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4" podUID="3a90bbcb-2fbc-4058-a5f0-3b78bde798bc"
Oct 30 20:00:14.616678 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:14.616652    5408 scope.go:122] "RemoveContainer" containerID="f2a67c60f85175176b0ce7785f86c91e1845b609517a5cc4fcbf26c26efb9421"
Oct 30 20:00:14.617077 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:14.617009    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk_odh-ai-gateway-infra(ac08d47c-3b2b-4825-8f49-c8283b06ad63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk" podUID="ac08d47c-3b2b-4825-8f49-c8283b06ad63"
Oct 30 20:00:14.618406 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:14.618386    5408 scope.go:122] "RemoveContainer" containerID="06f917e2f8441f24e923a47ffcf177b6b7b1989d5b236d59b8a66fa829a4b646"
Oct 30 20:00:14.618701 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:14.618678    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk_odh-ai-gateway-infra(21b60350-a468-4dec-afb8-c1f67c3939e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk" podUID="21b60350-a468-4dec-afb8-c1f67c3939e5"
Oct 30 20:00:19.723211 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:19.723172    5408 generic.go:356] "Generic (PLEG): container finished" podID="b88da0c0-5e7b-42a1-9742-f88cc062c2f2" containerID="a932727e4cdd0fb2793b7dd3da35f84c128f8acf152c3c76d5ce0a991ae03f13" exitCode=60
Oct 30 20:00:19.723654 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:19.723259    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8" event={"ID":"b88da0c0-5e7b-42a1-9742-f88cc062c2f2","Type":"ContainerDied","Data":"a932727e4cdd0fb2793b7dd3da35f84c128f8acf152c3c76d5ce0a991ae03f13"}
Oct 30 20:00:19.723654 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:19.723626    5408 scope.go:122] "RemoveContainer" containerID="a932727e4cdd0fb2793b7dd3da35f84c128f8acf152c3c76d5ce0a991ae03f13"
Oct 30 20:00:20.525560 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:20.525529    5408 scope.go:122] "RemoveContainer" containerID="d6c88fc9a037e06aff2f2b26646aecbf72775ff388f6385d576857ff6f202f47"
Oct 30 20:00:20.778111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:20.776593    5408 generic.go:356] "Generic (PLEG): container finished" podID="b88da0c0-5e7b-42a1-9742-f88cc062c2f2" containerID="6185b854fd9e4259dcd154e3adc99ca9176bb7a4d431e1e82ab500c48086849b" exitCode=60
Oct 30 20:00:20.778111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:20.776888    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8" event={"ID":"b88da0c0-5e7b-42a1-9742-f88cc062c2f2","Type":"ContainerDied","Data":"6185b854fd9e4259dcd154e3adc99ca9176bb7a4d431e1e82ab500c48086849b"}
Oct 30 20:00:20.778111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:20.776964    5408 scope.go:122] "RemoveContainer" containerID="a932727e4cdd0fb2793b7dd3da35f84c128f8acf152c3c76d5ce0a991ae03f13"
Oct 30 20:00:20.778111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:20.777643    5408 scope.go:122] "RemoveContainer" containerID="6185b854fd9e4259dcd154e3adc99ca9176bb7a4d431e1e82ab500c48086849b"
Oct 30 20:00:20.780909 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:20.778910    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8_odh-ai-gateway-infra(b88da0c0-5e7b-42a1-9742-f88cc062c2f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8" podUID="b88da0c0-5e7b-42a1-9742-f88cc062c2f2"
Oct 30 20:00:21.552753 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:21.546962    5408 scope.go:122] "RemoveContainer" containerID="05a7a29badcc2eec02d7caac041b44845eeb851c72a2a5914e22483562e73f4c"
Oct 30 20:00:21.552753 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:21.547760    5408 scope.go:122] "RemoveContainer" containerID="0e2e4ac817adb750b38561d2debbd68bd07766df9c7c6c8b60d7a557c8b50c4e"
Oct 30 20:00:21.812101 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:21.812069    5408 scope.go:122] "RemoveContainer" containerID="6185b854fd9e4259dcd154e3adc99ca9176bb7a4d431e1e82ab500c48086849b"
Oct 30 20:00:21.813785 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:21.813753    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8_odh-ai-gateway-infra(b88da0c0-5e7b-42a1-9742-f88cc062c2f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8" podUID="b88da0c0-5e7b-42a1-9742-f88cc062c2f2"
Oct 30 20:00:21.817293 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:21.817268    5408 generic.go:356] "Generic (PLEG): container finished" podID="53f59233-5357-4386-9f05-2acfa34200dd" containerID="92e2ef794f6593abb6ba9d7560f4f53d5cb87024e8e6006d78b87ac247358aa2" exitCode=60
Oct 30 20:00:21.817516 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:21.817499    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42" event={"ID":"53f59233-5357-4386-9f05-2acfa34200dd","Type":"ContainerDied","Data":"92e2ef794f6593abb6ba9d7560f4f53d5cb87024e8e6006d78b87ac247358aa2"}
Oct 30 20:00:21.817671 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:21.817658    5408 scope.go:122] "RemoveContainer" containerID="d6c88fc9a037e06aff2f2b26646aecbf72775ff388f6385d576857ff6f202f47"
Oct 30 20:00:21.818432 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:21.818415    5408 scope.go:122] "RemoveContainer" containerID="92e2ef794f6593abb6ba9d7560f4f53d5cb87024e8e6006d78b87ac247358aa2"
Oct 30 20:00:21.819226 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:21.819199    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42_odh-ai-gateway-infra(53f59233-5357-4386-9f05-2acfa34200dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42" podUID="53f59233-5357-4386-9f05-2acfa34200dd"
Oct 30 20:00:22.525313 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:22.525281    5408 scope.go:122] "RemoveContainer" containerID="3b2c357044a9a0a2c250a7ee7caa239d6a867d441536b5aa533489158067d77f"
Oct 30 20:00:22.525503 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:22.525484    5408 scope.go:122] "RemoveContainer" containerID="01cdd02cd66e6487c52af0662fe764ff3be7f729cfaa1725c74254f29cf5e14c"
Oct 30 20:00:22.827459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:22.827432    5408 generic.go:356] "Generic (PLEG): container finished" podID="8212a072-9983-47fd-9f83-08cd1a34747c" containerID="c85ec2b7ee21a9b4b4981da0240946135398cc48d30871359ac607d9e2f5bfd9" exitCode=60
Oct 30 20:00:22.827825 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:22.827508    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8" event={"ID":"8212a072-9983-47fd-9f83-08cd1a34747c","Type":"ContainerDied","Data":"c85ec2b7ee21a9b4b4981da0240946135398cc48d30871359ac607d9e2f5bfd9"}
Oct 30 20:00:22.827825 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:22.827575    5408 scope.go:122] "RemoveContainer" containerID="05a7a29badcc2eec02d7caac041b44845eeb851c72a2a5914e22483562e73f4c"
Oct 30 20:00:22.828065 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:22.828015    5408 scope.go:122] "RemoveContainer" containerID="c85ec2b7ee21a9b4b4981da0240946135398cc48d30871359ac607d9e2f5bfd9"
Oct 30 20:00:22.828437 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:22.828413    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8_odh-ai-gateway-infra(8212a072-9983-47fd-9f83-08cd1a34747c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8" podUID="8212a072-9983-47fd-9f83-08cd1a34747c"
Oct 30 20:00:22.831400 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:22.831380    5408 generic.go:356] "Generic (PLEG): container finished" podID="29d94ec1-c6ac-42ad-8244-2df9bf785f01" containerID="d988df1228b25a1d25e0429671695e8856365712833bcc78b77708d560025f8b" exitCode=60
Oct 30 20:00:22.831485 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:22.831446    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr" event={"ID":"29d94ec1-c6ac-42ad-8244-2df9bf785f01","Type":"ContainerDied","Data":"d988df1228b25a1d25e0429671695e8856365712833bcc78b77708d560025f8b"}
Oct 30 20:00:22.831877 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:22.831859    5408 scope.go:122] "RemoveContainer" containerID="d988df1228b25a1d25e0429671695e8856365712833bcc78b77708d560025f8b"
Oct 30 20:00:22.832257 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:22.832231    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr_odh-ai-gateway-infra(29d94ec1-c6ac-42ad-8244-2df9bf785f01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr" podUID="29d94ec1-c6ac-42ad-8244-2df9bf785f01"
Oct 30 20:00:23.037894 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:23.035395    5408 scope.go:122] "RemoveContainer" containerID="0e2e4ac817adb750b38561d2debbd68bd07766df9c7c6c8b60d7a557c8b50c4e"
Oct 30 20:00:23.355316 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:23.354913    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42"]
Oct 30 20:00:23.527486 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:23.527449    5408 scope.go:122] "RemoveContainer" containerID="06f917e2f8441f24e923a47ffcf177b6b7b1989d5b236d59b8a66fa829a4b646"
Oct 30 20:00:23.528220 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:23.528191    5408 scope.go:122] "RemoveContainer" containerID="94a6354dd54e9a2c74866ba37fb12d8c9ab15adb578b3c21c645b05ba3a9e5db"
Oct 30 20:00:23.626418 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:23.626386    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42"
Oct 30 20:00:23.742725 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:23.742695    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f59233-5357-4386-9f05-2acfa34200dd-kube-api-access-9cd9l\" (UniqueName: \"kubernetes.io/projected/53f59233-5357-4386-9f05-2acfa34200dd-kube-api-access-9cd9l\") pod \"53f59233-5357-4386-9f05-2acfa34200dd\" (UID: \"53f59233-5357-4386-9f05-2acfa34200dd\") "
Oct 30 20:00:23.742881 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:23.742796    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53f59233-5357-4386-9f05-2acfa34200dd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f59233-5357-4386-9f05-2acfa34200dd-maas-api-service-ca\") pod \"53f59233-5357-4386-9f05-2acfa34200dd\" (UID: \"53f59233-5357-4386-9f05-2acfa34200dd\") "
Oct 30 20:00:23.743196 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:23.743167    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f59233-5357-4386-9f05-2acfa34200dd-maas-api-service-ca" pod "53f59233-5357-4386-9f05-2acfa34200dd" (UID: "53f59233-5357-4386-9f05-2acfa34200dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:23.743531 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:23.743511    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f59233-5357-4386-9f05-2acfa34200dd-maas-api-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:23.805436 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:23.805388    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f59233-5357-4386-9f05-2acfa34200dd-kube-api-access-9cd9l" pod "53f59233-5357-4386-9f05-2acfa34200dd" (UID: "53f59233-5357-4386-9f05-2acfa34200dd"). InnerVolumeSpecName "kube-api-access-9cd9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:23.866118 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:23.864929    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cd9l\" (UniqueName: \"kubernetes.io/projected/53f59233-5357-4386-9f05-2acfa34200dd-kube-api-access-9cd9l\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:23.960521 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:23.960489    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8"]
Oct 30 20:00:23.992950 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:23.992907    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42" event={"ID":"53f59233-5357-4386-9f05-2acfa34200dd","Type":"ContainerDied","Data":"4cc30c15d2db7bbb3425a60164d6671dee6486114e138b17681a65b7b25d0b5d"}
Oct 30 20:00:23.993131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:23.992931    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42"
Oct 30 20:00:23.993131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:23.992988    5408 scope.go:122] "RemoveContainer" containerID="92e2ef794f6593abb6ba9d7560f4f53d5cb87024e8e6006d78b87ac247358aa2"
Oct 30 20:00:23.993937 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:23.993913    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr"]
Oct 30 20:00:24.043013 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.042376    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62" event={"ID":"943fe46f-f061-4c37-80e8-0958e7e145c3","Type":"ContainerStarted","Data":"4cf54a701c17b16ebdb31476ba81ac5b19ee8edab0bd3df814704916aaf8ff14"}
Oct 30 20:00:24.043185 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.043163    5408 scope.go:122] "RemoveContainer" containerID="4cf54a701c17b16ebdb31476ba81ac5b19ee8edab0bd3df814704916aaf8ff14"
Oct 30 20:00:24.046612 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.046587    5408 generic.go:356] "Generic (PLEG): container finished" podID="da1e7396-fe2d-4cb7-a019-0f6323f3b151" containerID="1e677553b06be4a69b3d6edca6c8a9072525ff274008da4fc40d0a49ae0c86fc" exitCode=60
Oct 30 20:00:24.047274 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:24.046830    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62_odh-ai-gateway-infra(943fe46f-f061-4c37-80e8-0958e7e145c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62" podUID="943fe46f-f061-4c37-80e8-0958e7e145c3"
Oct 30 20:00:24.048090 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.048064    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb" event={"ID":"da1e7396-fe2d-4cb7-a019-0f6323f3b151","Type":"ContainerDied","Data":"1e677553b06be4a69b3d6edca6c8a9072525ff274008da4fc40d0a49ae0c86fc"}
Oct 30 20:00:24.049605 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.048598    5408 scope.go:122] "RemoveContainer" containerID="1e677553b06be4a69b3d6edca6c8a9072525ff274008da4fc40d0a49ae0c86fc"
Oct 30 20:00:24.049605 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:24.049202    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb_odh-ai-gateway-infra(da1e7396-fe2d-4cb7-a019-0f6323f3b151)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb" podUID="da1e7396-fe2d-4cb7-a019-0f6323f3b151"
Oct 30 20:00:24.100640 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.100615    5408 scope.go:122] "RemoveContainer" containerID="3b2c357044a9a0a2c250a7ee7caa239d6a867d441536b5aa533489158067d77f"
Oct 30 20:00:24.227928 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.227811    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42"]
Oct 30 20:00:24.235403 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.235377    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c4c072-29889840-qbj42"]
Oct 30 20:00:24.249088 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.248001    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8"
Oct 30 20:00:24.279609 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.279578    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr"
Oct 30 20:00:24.286135 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.285944    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8212a072-9983-47fd-9f83-08cd1a34747c-kube-api-access-mxpmm\" (UniqueName: \"kubernetes.io/projected/8212a072-9983-47fd-9f83-08cd1a34747c-kube-api-access-mxpmm\") pod \"8212a072-9983-47fd-9f83-08cd1a34747c\" (UID: \"8212a072-9983-47fd-9f83-08cd1a34747c\") "
Oct 30 20:00:24.287066 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.286939    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8212a072-9983-47fd-9f83-08cd1a34747c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8212a072-9983-47fd-9f83-08cd1a34747c-maas-api-service-ca\") pod \"8212a072-9983-47fd-9f83-08cd1a34747c\" (UID: \"8212a072-9983-47fd-9f83-08cd1a34747c\") "
Oct 30 20:00:24.287236 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.287211    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d94ec1-c6ac-42ad-8244-2df9bf785f01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29d94ec1-c6ac-42ad-8244-2df9bf785f01-maas-api-service-ca\") pod \"29d94ec1-c6ac-42ad-8244-2df9bf785f01\" (UID: \"29d94ec1-c6ac-42ad-8244-2df9bf785f01\") "
Oct 30 20:00:24.287366 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.287329    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d94ec1-c6ac-42ad-8244-2df9bf785f01-kube-api-access-kbn7w\" (UniqueName: \"kubernetes.io/projected/29d94ec1-c6ac-42ad-8244-2df9bf785f01-kube-api-access-kbn7w\") pod \"29d94ec1-c6ac-42ad-8244-2df9bf785f01\" (UID: \"29d94ec1-c6ac-42ad-8244-2df9bf785f01\") "
Oct 30 20:00:24.287431 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.287374    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8212a072-9983-47fd-9f83-08cd1a34747c-maas-api-service-ca" pod "8212a072-9983-47fd-9f83-08cd1a34747c" (UID: "8212a072-9983-47fd-9f83-08cd1a34747c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:24.287621 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.287600    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d94ec1-c6ac-42ad-8244-2df9bf785f01-maas-api-service-ca" pod "29d94ec1-c6ac-42ad-8244-2df9bf785f01" (UID: "29d94ec1-c6ac-42ad-8244-2df9bf785f01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:24.288099 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.288017    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29d94ec1-c6ac-42ad-8244-2df9bf785f01-maas-api-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:24.288099 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.288082    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8212a072-9983-47fd-9f83-08cd1a34747c-maas-api-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:24.288683 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.288660    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8212a072-9983-47fd-9f83-08cd1a34747c-kube-api-access-mxpmm" pod "8212a072-9983-47fd-9f83-08cd1a34747c" (UID: "8212a072-9983-47fd-9f83-08cd1a34747c"). InnerVolumeSpecName "kube-api-access-mxpmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:24.289756 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.289719    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d94ec1-c6ac-42ad-8244-2df9bf785f01-kube-api-access-kbn7w" pod "29d94ec1-c6ac-42ad-8244-2df9bf785f01" (UID: "29d94ec1-c6ac-42ad-8244-2df9bf785f01"). InnerVolumeSpecName "kube-api-access-kbn7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:24.390284 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.390204    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbn7w\" (UniqueName: \"kubernetes.io/projected/29d94ec1-c6ac-42ad-8244-2df9bf785f01-kube-api-access-kbn7w\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:24.390284 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:24.390233    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxpmm\" (UniqueName: \"kubernetes.io/projected/8212a072-9983-47fd-9f83-08cd1a34747c-kube-api-access-mxpmm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:25.056866 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.056839    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8"
Oct 30 20:00:25.057321 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.056875    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8" event={"ID":"8212a072-9983-47fd-9f83-08cd1a34747c","Type":"ContainerDied","Data":"0266a1be98f40be15697057b65331070f0290cd74f2bae1281d79bd967b760e8"}
Oct 30 20:00:25.057321 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.056935    5408 scope.go:122] "RemoveContainer" containerID="c85ec2b7ee21a9b4b4981da0240946135398cc48d30871359ac607d9e2f5bfd9"
Oct 30 20:00:25.059371 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.059342    5408 generic.go:356] "Generic (PLEG): container finished" podID="f940d6ee-3ee6-4484-8689-5ece47e6c1ab" containerID="ebe9e1d076101828ab72f7e4600ed2502ba7607508ac6df5a8ece7f88a970215" exitCode=60
Oct 30 20:00:25.059496 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.059410    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz" event={"ID":"f940d6ee-3ee6-4484-8689-5ece47e6c1ab","Type":"ContainerDied","Data":"ebe9e1d076101828ab72f7e4600ed2502ba7607508ac6df5a8ece7f88a970215"}
Oct 30 20:00:25.059897 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.059880    5408 scope.go:122] "RemoveContainer" containerID="ebe9e1d076101828ab72f7e4600ed2502ba7607508ac6df5a8ece7f88a970215"
Oct 30 20:00:25.060316 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:25.060284    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz_odh-ai-gateway-infra(f940d6ee-3ee6-4484-8689-5ece47e6c1ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz" podUID="f940d6ee-3ee6-4484-8689-5ece47e6c1ab"
Oct 30 20:00:25.062481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.062431    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr"
Oct 30 20:00:25.062665 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.062496    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr" event={"ID":"29d94ec1-c6ac-42ad-8244-2df9bf785f01","Type":"ContainerDied","Data":"a7c9e01eb09e0866398e3732da0700a0d3d2af131b3f0d39af38a41c544af007"}
Oct 30 20:00:25.065359 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.065334    5408 generic.go:356] "Generic (PLEG): container finished" podID="943fe46f-f061-4c37-80e8-0958e7e145c3" containerID="4cf54a701c17b16ebdb31476ba81ac5b19ee8edab0bd3df814704916aaf8ff14" exitCode=60
Oct 30 20:00:25.065463 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.065412    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62" event={"ID":"943fe46f-f061-4c37-80e8-0958e7e145c3","Type":"ContainerDied","Data":"4cf54a701c17b16ebdb31476ba81ac5b19ee8edab0bd3df814704916aaf8ff14"}
Oct 30 20:00:25.066013 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.065879    5408 scope.go:122] "RemoveContainer" containerID="4cf54a701c17b16ebdb31476ba81ac5b19ee8edab0bd3df814704916aaf8ff14"
Oct 30 20:00:25.066241 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:25.066216    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62_odh-ai-gateway-infra(943fe46f-f061-4c37-80e8-0958e7e145c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62" podUID="943fe46f-f061-4c37-80e8-0958e7e145c3"
Oct 30 20:00:25.069548 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.069528    5408 generic.go:356] "Generic (PLEG): container finished" podID="21b60350-a468-4dec-afb8-c1f67c3939e5" containerID="163b82f34233151e30f709306310d1155863ec8fa2361f0a766f9d299c88ddbe" exitCode=60
Oct 30 20:00:25.069638 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.069570    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk" event={"ID":"21b60350-a468-4dec-afb8-c1f67c3939e5","Type":"ContainerDied","Data":"163b82f34233151e30f709306310d1155863ec8fa2361f0a766f9d299c88ddbe"}
Oct 30 20:00:25.069937 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.069921    5408 scope.go:122] "RemoveContainer" containerID="163b82f34233151e30f709306310d1155863ec8fa2361f0a766f9d299c88ddbe"
Oct 30 20:00:25.070232 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:25.070211    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk_odh-ai-gateway-infra(21b60350-a468-4dec-afb8-c1f67c3939e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk" podUID="21b60350-a468-4dec-afb8-c1f67c3939e5"
Oct 30 20:00:25.070618 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.070580    5408 scope.go:122] "RemoveContainer" containerID="94a6354dd54e9a2c74866ba37fb12d8c9ab15adb578b3c21c645b05ba3a9e5db"
Oct 30 20:00:25.163049 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.154457    5408 scope.go:122] "RemoveContainer" containerID="d988df1228b25a1d25e0429671695e8856365712833bcc78b77708d560025f8b"
Oct 30 20:00:25.216175 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.202090    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62"]
Oct 30 20:00:25.238745 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.238578    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb"]
Oct 30 20:00:25.245593 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.245572    5408 scope.go:122] "RemoveContainer" containerID="01cdd02cd66e6487c52af0662fe764ff3be7f729cfaa1725c74254f29cf5e14c"
Oct 30 20:00:25.250638 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.250613    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8"]
Oct 30 20:00:25.254478 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.254457    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a1d885-29889840-t5cq8"]
Oct 30 20:00:25.297136 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.296198    5408 scope.go:122] "RemoveContainer" containerID="06f917e2f8441f24e923a47ffcf177b6b7b1989d5b236d59b8a66fa829a4b646"
Oct 30 20:00:25.329103 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.323699    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr"]
Oct 30 20:00:25.394965 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.375086    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-98a0d8a9-29889840-94qlr"]
Oct 30 20:00:25.530009 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.529982    5408 scope.go:122] "RemoveContainer" containerID="506bc9c98f62f9e6a2b6aa5bcd9592b4f8e9a58b02ed5bc272d2b0949112b9f7"
Oct 30 20:00:25.530429 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.530403    5408 scope.go:122] "RemoveContainer" containerID="d3f4b33c95b045068e438dc4f0ce3189861db2f54dd814579bffb66c069593f1"
Oct 30 20:00:25.540337 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.540318    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb"
Oct 30 20:00:25.540461 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.540436    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29d94ec1-c6ac-42ad-8244-2df9bf785f01" path="/var/lib/kubelet/pods/29d94ec1-c6ac-42ad-8244-2df9bf785f01/volumes"
Oct 30 20:00:25.541135 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.541116    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53f59233-5357-4386-9f05-2acfa34200dd" path="/var/lib/kubelet/pods/53f59233-5357-4386-9f05-2acfa34200dd/volumes"
Oct 30 20:00:25.541693 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.541676    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8212a072-9983-47fd-9f83-08cd1a34747c" path="/var/lib/kubelet/pods/8212a072-9983-47fd-9f83-08cd1a34747c/volumes"
Oct 30 20:00:25.647239 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.647208    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da1e7396-fe2d-4cb7-a019-0f6323f3b151-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1e7396-fe2d-4cb7-a019-0f6323f3b151-maas-api-service-ca\") pod \"da1e7396-fe2d-4cb7-a019-0f6323f3b151\" (UID: \"da1e7396-fe2d-4cb7-a019-0f6323f3b151\") "
Oct 30 20:00:25.647375 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.647318    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da1e7396-fe2d-4cb7-a019-0f6323f3b151-kube-api-access-d4t6c\" (UniqueName: \"kubernetes.io/projected/da1e7396-fe2d-4cb7-a019-0f6323f3b151-kube-api-access-d4t6c\") pod \"da1e7396-fe2d-4cb7-a019-0f6323f3b151\" (UID: \"da1e7396-fe2d-4cb7-a019-0f6323f3b151\") "
Oct 30 20:00:25.647689 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.647665    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da1e7396-fe2d-4cb7-a019-0f6323f3b151-maas-api-service-ca" pod "da1e7396-fe2d-4cb7-a019-0f6323f3b151" (UID: "da1e7396-fe2d-4cb7-a019-0f6323f3b151"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:25.648272 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.648253    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1e7396-fe2d-4cb7-a019-0f6323f3b151-maas-api-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:25.703936 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.703889    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1e7396-fe2d-4cb7-a019-0f6323f3b151-kube-api-access-d4t6c" pod "da1e7396-fe2d-4cb7-a019-0f6323f3b151" (UID: "da1e7396-fe2d-4cb7-a019-0f6323f3b151"). InnerVolumeSpecName "kube-api-access-d4t6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:25.749731 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:25.749698    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4t6c\" (UniqueName: \"kubernetes.io/projected/da1e7396-fe2d-4cb7-a019-0f6323f3b151-kube-api-access-d4t6c\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:26.081634 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.081593    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78" event={"ID":"39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03","Type":"ContainerStarted","Data":"9df4b065544dcc460cee120abfe8fc5662ceb6266f8bff096e2f8a055f62c508"}
Oct 30 20:00:26.086999 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.086964    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4" event={"ID":"3a90bbcb-2fbc-4058-a5f0-3b78bde798bc","Type":"ContainerStarted","Data":"91cb944932f8364657f64e12d337dcef7b64869c2eefbe2fe35864b43161245e"}
Oct 30 20:00:26.090924 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.090892    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb"
Oct 30 20:00:26.091078 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.090904    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb" event={"ID":"da1e7396-fe2d-4cb7-a019-0f6323f3b151","Type":"ContainerDied","Data":"dba10e77bd442d7edae2c42e53474dbb120b8f64cd9c2f787d6c49e9761ef61d"}
Oct 30 20:00:26.091187 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.091104    5408 scope.go:122] "RemoveContainer" containerID="1e677553b06be4a69b3d6edca6c8a9072525ff274008da4fc40d0a49ae0c86fc"
Oct 30 20:00:26.095975 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.095954    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz"]
Oct 30 20:00:26.197576 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.197542    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb"]
Oct 30 20:00:26.204388 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.204355    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-dd1b04-29889840-7gnpb"]
Oct 30 20:00:26.230625 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.230596    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk"]
Oct 30 20:00:26.326118 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.325547    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62"
Oct 30 20:00:26.362548 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.362514    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/943fe46f-f061-4c37-80e8-0958e7e145c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/943fe46f-f061-4c37-80e8-0958e7e145c3-maas-api-service-ca\") pod \"943fe46f-f061-4c37-80e8-0958e7e145c3\" (UID: \"943fe46f-f061-4c37-80e8-0958e7e145c3\") "
Oct 30 20:00:26.362884 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.362862    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/943fe46f-f061-4c37-80e8-0958e7e145c3-kube-api-access-qgfhm\" (UniqueName: \"kubernetes.io/projected/943fe46f-f061-4c37-80e8-0958e7e145c3-kube-api-access-qgfhm\") pod \"943fe46f-f061-4c37-80e8-0958e7e145c3\" (UID: \"943fe46f-f061-4c37-80e8-0958e7e145c3\") "
Oct 30 20:00:26.362976 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.362949    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943fe46f-f061-4c37-80e8-0958e7e145c3-maas-api-service-ca" pod "943fe46f-f061-4c37-80e8-0958e7e145c3" (UID: "943fe46f-f061-4c37-80e8-0958e7e145c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:26.364303 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.364268    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/943fe46f-f061-4c37-80e8-0958e7e145c3-maas-api-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:26.373331 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.373303    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943fe46f-f061-4c37-80e8-0958e7e145c3-kube-api-access-qgfhm" pod "943fe46f-f061-4c37-80e8-0958e7e145c3" (UID: "943fe46f-f061-4c37-80e8-0958e7e145c3"). InnerVolumeSpecName "kube-api-access-qgfhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:26.386011 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.385981    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz"
Oct 30 20:00:26.422121 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.421940    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk"
Oct 30 20:00:26.466900 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.466816    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b60350-a468-4dec-afb8-c1f67c3939e5-kube-api-access-n9bbr\" (UniqueName: \"kubernetes.io/projected/21b60350-a468-4dec-afb8-c1f67c3939e5-kube-api-access-n9bbr\") pod \"21b60350-a468-4dec-afb8-c1f67c3939e5\" (UID: \"21b60350-a468-4dec-afb8-c1f67c3939e5\") "
Oct 30 20:00:26.467109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.466916    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b60350-a468-4dec-afb8-c1f67c3939e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b60350-a468-4dec-afb8-c1f67c3939e5-maas-api-service-ca\") pod \"21b60350-a468-4dec-afb8-c1f67c3939e5\" (UID: \"21b60350-a468-4dec-afb8-c1f67c3939e5\") "
Oct 30 20:00:26.467109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.467089    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f940d6ee-3ee6-4484-8689-5ece47e6c1ab-kube-api-access-mcm5r\" (UniqueName: \"kubernetes.io/projected/f940d6ee-3ee6-4484-8689-5ece47e6c1ab-kube-api-access-mcm5r\") pod \"f940d6ee-3ee6-4484-8689-5ece47e6c1ab\" (UID: \"f940d6ee-3ee6-4484-8689-5ece47e6c1ab\") "
Oct 30 20:00:26.467243 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.467140    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f940d6ee-3ee6-4484-8689-5ece47e6c1ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f940d6ee-3ee6-4484-8689-5ece47e6c1ab-maas-api-service-ca\") pod \"f940d6ee-3ee6-4484-8689-5ece47e6c1ab\" (UID: \"f940d6ee-3ee6-4484-8689-5ece47e6c1ab\") "
Oct 30 20:00:26.468088 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.468058    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b60350-a468-4dec-afb8-c1f67c3939e5-maas-api-service-ca" pod "21b60350-a468-4dec-afb8-c1f67c3939e5" (UID: "21b60350-a468-4dec-afb8-c1f67c3939e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:26.468211 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.468125    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgfhm\" (UniqueName: \"kubernetes.io/projected/943fe46f-f061-4c37-80e8-0958e7e145c3-kube-api-access-qgfhm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:26.468763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.468734    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f940d6ee-3ee6-4484-8689-5ece47e6c1ab-maas-api-service-ca" pod "f940d6ee-3ee6-4484-8689-5ece47e6c1ab" (UID: "f940d6ee-3ee6-4484-8689-5ece47e6c1ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:26.472574 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.472549    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f940d6ee-3ee6-4484-8689-5ece47e6c1ab-kube-api-access-mcm5r" pod "f940d6ee-3ee6-4484-8689-5ece47e6c1ab" (UID: "f940d6ee-3ee6-4484-8689-5ece47e6c1ab"). InnerVolumeSpecName "kube-api-access-mcm5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:26.473862 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.473835    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b60350-a468-4dec-afb8-c1f67c3939e5-kube-api-access-n9bbr" pod "21b60350-a468-4dec-afb8-c1f67c3939e5" (UID: "21b60350-a468-4dec-afb8-c1f67c3939e5"). InnerVolumeSpecName "kube-api-access-n9bbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:26.525457 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.525391    5408 scope.go:122] "RemoveContainer" containerID="f2a67c60f85175176b0ce7785f86c91e1845b609517a5cc4fcbf26c26efb9421"
Oct 30 20:00:26.569761 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.569737    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9bbr\" (UniqueName: \"kubernetes.io/projected/21b60350-a468-4dec-afb8-c1f67c3939e5-kube-api-access-n9bbr\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:26.569869 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.569765    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b60350-a468-4dec-afb8-c1f67c3939e5-maas-api-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:26.569869 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.569781    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcm5r\" (UniqueName: \"kubernetes.io/projected/f940d6ee-3ee6-4484-8689-5ece47e6c1ab-kube-api-access-mcm5r\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:26.569869 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:26.569795    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f940d6ee-3ee6-4484-8689-5ece47e6c1ab-maas-api-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:27.134846 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.134015    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78"]
Oct 30 20:00:27.150979 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.150943    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk" event={"ID":"21b60350-a468-4dec-afb8-c1f67c3939e5","Type":"ContainerDied","Data":"088830f89045681e68d59b87dea93b613ac3d133de85eeaebe6c6c6352355d73"}
Oct 30 20:00:27.151118 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.150958    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk"
Oct 30 20:00:27.151179 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.151016    5408 scope.go:122] "RemoveContainer" containerID="163b82f34233151e30f709306310d1155863ec8fa2361f0a766f9d299c88ddbe"
Oct 30 20:00:27.156140 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.154120    5408 generic.go:356] "Generic (PLEG): container finished" podID="39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03" containerID="9df4b065544dcc460cee120abfe8fc5662ceb6266f8bff096e2f8a055f62c508" exitCode=60
Oct 30 20:00:27.156140 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.154226    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78" event={"ID":"39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03","Type":"ContainerDied","Data":"9df4b065544dcc460cee120abfe8fc5662ceb6266f8bff096e2f8a055f62c508"}
Oct 30 20:00:27.157877 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.157854    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz"
Oct 30 20:00:27.158832 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.158808    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz" event={"ID":"f940d6ee-3ee6-4484-8689-5ece47e6c1ab","Type":"ContainerDied","Data":"6755e8eefcebb0f28101e183e07470befe298b5eee5587430d8578bbc41f4698"}
Oct 30 20:00:27.164318 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.164287    5408 generic.go:356] "Generic (PLEG): container finished" podID="3a90bbcb-2fbc-4058-a5f0-3b78bde798bc" containerID="91cb944932f8364657f64e12d337dcef7b64869c2eefbe2fe35864b43161245e" exitCode=60
Oct 30 20:00:27.164452 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.164443    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4" event={"ID":"3a90bbcb-2fbc-4058-a5f0-3b78bde798bc","Type":"ContainerDied","Data":"91cb944932f8364657f64e12d337dcef7b64869c2eefbe2fe35864b43161245e"}
Oct 30 20:00:27.164932 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.164913    5408 scope.go:122] "RemoveContainer" containerID="91cb944932f8364657f64e12d337dcef7b64869c2eefbe2fe35864b43161245e"
Oct 30 20:00:27.165470 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:27.165443    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4_odh-ai-gateway-infra(3a90bbcb-2fbc-4058-a5f0-3b78bde798bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4" podUID="3a90bbcb-2fbc-4058-a5f0-3b78bde798bc"
Oct 30 20:00:27.172729 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.172694    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62" event={"ID":"943fe46f-f061-4c37-80e8-0958e7e145c3","Type":"ContainerDied","Data":"69239af9e2e4de96b16b746bd29f58c3704d8a16c6a6b41f51352912d3510b80"}
Oct 30 20:00:27.172812 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.172793    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62"
Oct 30 20:00:27.175397 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.175374    5408 scope.go:122] "RemoveContainer" containerID="506bc9c98f62f9e6a2b6aa5bcd9592b4f8e9a58b02ed5bc272d2b0949112b9f7"
Oct 30 20:00:27.197600 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.197573    5408 scope.go:122] "RemoveContainer" containerID="ebe9e1d076101828ab72f7e4600ed2502ba7607508ac6df5a8ece7f88a970215"
Oct 30 20:00:27.216928 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.216903    5408 scope.go:122] "RemoveContainer" containerID="d3f4b33c95b045068e438dc4f0ce3189861db2f54dd814579bffb66c069593f1"
Oct 30 20:00:27.228252 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.228225    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4"]
Oct 30 20:00:27.233824 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.233801    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz"]
Oct 30 20:00:27.238113 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.235801    5408 scope.go:122] "RemoveContainer" containerID="4cf54a701c17b16ebdb31476ba81ac5b19ee8edab0bd3df814704916aaf8ff14"
Oct 30 20:00:27.243041 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.243008    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a8de3b-29889840-smlkz"]
Oct 30 20:00:27.271110 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.271085    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62"]
Oct 30 20:00:27.285476 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.285423    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d5e3e5-29889840-6zg62"]
Oct 30 20:00:27.356737 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.355006    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk"]
Oct 30 20:00:27.359190 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.359166    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b0d13a-29889840-rgdxk"]
Oct 30 20:00:27.398695 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.398671    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78"
Oct 30 20:00:27.443092 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.443053    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03-maas-api-service-ca\") pod \"39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03\" (UID: \"39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03\") "
Oct 30 20:00:27.443332 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.443309    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03-maas-api-service-ca" pod "39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03" (UID: "39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:27.443413 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.443355    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03-kube-api-access-t94rv\" (UniqueName: \"kubernetes.io/projected/39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03-kube-api-access-t94rv\") pod \"39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03\" (UID: \"39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03\") "
Oct 30 20:00:27.444095 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.444073    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03-maas-api-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:27.446002 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.445972    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03-kube-api-access-t94rv" pod "39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03" (UID: "39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03"). InnerVolumeSpecName "kube-api-access-t94rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:27.534282 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.534253    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b60350-a468-4dec-afb8-c1f67c3939e5" path="/var/lib/kubelet/pods/21b60350-a468-4dec-afb8-c1f67c3939e5/volumes"
Oct 30 20:00:27.534815 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.534794    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="943fe46f-f061-4c37-80e8-0958e7e145c3" path="/var/lib/kubelet/pods/943fe46f-f061-4c37-80e8-0958e7e145c3/volumes"
Oct 30 20:00:27.535281 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.535266    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da1e7396-fe2d-4cb7-a019-0f6323f3b151" path="/var/lib/kubelet/pods/da1e7396-fe2d-4cb7-a019-0f6323f3b151/volumes"
Oct 30 20:00:27.536010 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.535996    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f940d6ee-3ee6-4484-8689-5ece47e6c1ab" path="/var/lib/kubelet/pods/f940d6ee-3ee6-4484-8689-5ece47e6c1ab/volumes"
Oct 30 20:00:27.545217 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:27.545198    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t94rv\" (UniqueName: \"kubernetes.io/projected/39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03-kube-api-access-t94rv\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:28.266135 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:28.266018    5408 generic.go:356] "Generic (PLEG): container finished" podID="ac08d47c-3b2b-4825-8f49-c8283b06ad63" containerID="f31df666ba086db648a3fda0099fcd02d4600d8b1299dc6bf133d9de673add45" exitCode=60
Oct 30 20:00:28.266519 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:28.266220    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk" event={"ID":"ac08d47c-3b2b-4825-8f49-c8283b06ad63","Type":"ContainerDied","Data":"f31df666ba086db648a3fda0099fcd02d4600d8b1299dc6bf133d9de673add45"}
Oct 30 20:00:28.266519 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:28.266271    5408 scope.go:122] "RemoveContainer" containerID="f2a67c60f85175176b0ce7785f86c91e1845b609517a5cc4fcbf26c26efb9421"
Oct 30 20:00:28.266765 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:28.266742    5408 scope.go:122] "RemoveContainer" containerID="f31df666ba086db648a3fda0099fcd02d4600d8b1299dc6bf133d9de673add45"
Oct 30 20:00:28.267218 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:28.267190    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk_odh-ai-gateway-infra(ac08d47c-3b2b-4825-8f49-c8283b06ad63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk" podUID="ac08d47c-3b2b-4825-8f49-c8283b06ad63"
Oct 30 20:00:28.270548 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:28.270530    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78"
Oct 30 20:00:28.270634 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:28.270571    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78" event={"ID":"39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03","Type":"ContainerDied","Data":"a751eb30f5ba2659f58f26ae6d1bfb02bb33025d2552e80e881d3f521998c66e"}
Oct 30 20:00:28.282644 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:28.282608    5408 scope.go:122] "RemoveContainer" containerID="9df4b065544dcc460cee120abfe8fc5662ceb6266f8bff096e2f8a055f62c508"
Oct 30 20:00:28.308376 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:28.303791    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78"]
Oct 30 20:00:28.313363 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:28.313327    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5f933e-29889840-j7q78"]
Oct 30 20:00:28.569395 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:28.554699    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4"
Oct 30 20:00:28.631053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:28.629628    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a90bbcb-2fbc-4058-a5f0-3b78bde798bc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a90bbcb-2fbc-4058-a5f0-3b78bde798bc-maas-api-service-ca\") pod \"3a90bbcb-2fbc-4058-a5f0-3b78bde798bc\" (UID: \"3a90bbcb-2fbc-4058-a5f0-3b78bde798bc\") "
Oct 30 20:00:28.631053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:28.629774    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a90bbcb-2fbc-4058-a5f0-3b78bde798bc-kube-api-access-v4cdk\" (UniqueName: \"kubernetes.io/projected/3a90bbcb-2fbc-4058-a5f0-3b78bde798bc-kube-api-access-v4cdk\") pod \"3a90bbcb-2fbc-4058-a5f0-3b78bde798bc\" (UID: \"3a90bbcb-2fbc-4058-a5f0-3b78bde798bc\") "
Oct 30 20:00:28.648099 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:28.643361    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a90bbcb-2fbc-4058-a5f0-3b78bde798bc-maas-api-service-ca" pod "3a90bbcb-2fbc-4058-a5f0-3b78bde798bc" (UID: "3a90bbcb-2fbc-4058-a5f0-3b78bde798bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:28.714214 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:28.714174    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a90bbcb-2fbc-4058-a5f0-3b78bde798bc-kube-api-access-v4cdk" pod "3a90bbcb-2fbc-4058-a5f0-3b78bde798bc" (UID: "3a90bbcb-2fbc-4058-a5f0-3b78bde798bc"). InnerVolumeSpecName "kube-api-access-v4cdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:28.757063 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:28.744641    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a90bbcb-2fbc-4058-a5f0-3b78bde798bc-maas-api-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:28.757063 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:28.744678    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4cdk\" (UniqueName: \"kubernetes.io/projected/3a90bbcb-2fbc-4058-a5f0-3b78bde798bc-kube-api-access-v4cdk\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:29.280986 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:29.280957    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4"
Oct 30 20:00:29.281449 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:29.280957    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4" event={"ID":"3a90bbcb-2fbc-4058-a5f0-3b78bde798bc","Type":"ContainerDied","Data":"2481b1ce898fd0cfa198872cd5a9afcf94e134fbf95beb77a4239bc96db47ece"}
Oct 30 20:00:29.281449 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:29.281148    5408 scope.go:122] "RemoveContainer" containerID="91cb944932f8364657f64e12d337dcef7b64869c2eefbe2fe35864b43161245e"
Oct 30 20:00:29.311956 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:29.311930    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4"]
Oct 30 20:00:29.319067 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:29.319044    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889840-56xv4"]
Oct 30 20:00:29.343888 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:29.343864    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk"]
Oct 30 20:00:29.535978 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:29.535950    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03" path="/var/lib/kubelet/pods/39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03/volumes"
Oct 30 20:00:29.536562 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:29.536538    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a90bbcb-2fbc-4058-a5f0-3b78bde798bc" path="/var/lib/kubelet/pods/3a90bbcb-2fbc-4058-a5f0-3b78bde798bc/volumes"
Oct 30 20:00:29.561995 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:29.561976    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk"
Oct 30 20:00:29.716275 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:29.711846    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac08d47c-3b2b-4825-8f49-c8283b06ad63-kube-api-access-kdd5f\" (UniqueName: \"kubernetes.io/projected/ac08d47c-3b2b-4825-8f49-c8283b06ad63-kube-api-access-kdd5f\") pod \"ac08d47c-3b2b-4825-8f49-c8283b06ad63\" (UID: \"ac08d47c-3b2b-4825-8f49-c8283b06ad63\") "
Oct 30 20:00:29.716275 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:29.712409    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac08d47c-3b2b-4825-8f49-c8283b06ad63-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac08d47c-3b2b-4825-8f49-c8283b06ad63-maas-api-service-ca\") pod \"ac08d47c-3b2b-4825-8f49-c8283b06ad63\" (UID: \"ac08d47c-3b2b-4825-8f49-c8283b06ad63\") "
Oct 30 20:00:29.716275 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:29.713199    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac08d47c-3b2b-4825-8f49-c8283b06ad63-maas-api-service-ca" pod "ac08d47c-3b2b-4825-8f49-c8283b06ad63" (UID: "ac08d47c-3b2b-4825-8f49-c8283b06ad63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:29.724319 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:29.716330    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac08d47c-3b2b-4825-8f49-c8283b06ad63-kube-api-access-kdd5f" pod "ac08d47c-3b2b-4825-8f49-c8283b06ad63" (UID: "ac08d47c-3b2b-4825-8f49-c8283b06ad63"). InnerVolumeSpecName "kube-api-access-kdd5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:29.814774 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:29.814677    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac08d47c-3b2b-4825-8f49-c8283b06ad63-maas-api-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:29.814774 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:29.814720    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdd5f\" (UniqueName: \"kubernetes.io/projected/ac08d47c-3b2b-4825-8f49-c8283b06ad63-kube-api-access-kdd5f\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:30.312556 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:30.312516    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk" event={"ID":"ac08d47c-3b2b-4825-8f49-c8283b06ad63","Type":"ContainerDied","Data":"510e4f41d4a6021b6464e77167742e48672e045c23c693160323aad16049afb5"}
Oct 30 20:00:30.313854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:30.313828    5408 scope.go:122] "RemoveContainer" containerID="f31df666ba086db648a3fda0099fcd02d4600d8b1299dc6bf133d9de673add45"
Oct 30 20:00:30.314272 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:30.313209    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk"
Oct 30 20:00:30.359181 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:30.359153    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk"]
Oct 30 20:00:30.373748 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:30.373721    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-278ffa-29889840-jd5mk"]
Oct 30 20:00:31.189089 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:31.189057    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb"]
Oct 30 20:00:31.205461 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:31.205432    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c"]
Oct 30 20:00:31.592055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:31.591461    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac08d47c-3b2b-4825-8f49-c8283b06ad63" path="/var/lib/kubelet/pods/ac08d47c-3b2b-4825-8f49-c8283b06ad63/volumes"
Oct 30 20:00:32.534982 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:32.534942    5408 scope.go:122] "RemoveContainer" containerID="6185b854fd9e4259dcd154e3adc99ca9176bb7a4d431e1e82ab500c48086849b"
Oct 30 20:00:33.345227 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:33.345186    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8" event={"ID":"b88da0c0-5e7b-42a1-9742-f88cc062c2f2","Type":"ContainerStarted","Data":"2359d4d3a7799bbacf8490dbbdb6c347f28a134809b0be95af1ecc3a7938060e"}
Oct 30 20:00:34.354476 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:34.354441    5408 generic.go:356] "Generic (PLEG): container finished" podID="b88da0c0-5e7b-42a1-9742-f88cc062c2f2" containerID="2359d4d3a7799bbacf8490dbbdb6c347f28a134809b0be95af1ecc3a7938060e" exitCode=60
Oct 30 20:00:34.354954 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:34.354511    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8" event={"ID":"b88da0c0-5e7b-42a1-9742-f88cc062c2f2","Type":"ContainerDied","Data":"2359d4d3a7799bbacf8490dbbdb6c347f28a134809b0be95af1ecc3a7938060e"}
Oct 30 20:00:34.354954 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:34.354578    5408 scope.go:122] "RemoveContainer" containerID="6185b854fd9e4259dcd154e3adc99ca9176bb7a4d431e1e82ab500c48086849b"
Oct 30 20:00:34.355131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:34.355020    5408 scope.go:122] "RemoveContainer" containerID="2359d4d3a7799bbacf8490dbbdb6c347f28a134809b0be95af1ecc3a7938060e"
Oct 30 20:00:34.355423 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:34.355403    5408 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8_odh-ai-gateway-infra(b88da0c0-5e7b-42a1-9742-f88cc062c2f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8" podUID="b88da0c0-5e7b-42a1-9742-f88cc062c2f2"
Oct 30 20:00:34.388579 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:34.388553    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8"]
Oct 30 20:00:35.658823 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:35.658801    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8"
Oct 30 20:00:35.789453 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:35.789426    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b88da0c0-5e7b-42a1-9742-f88cc062c2f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88da0c0-5e7b-42a1-9742-f88cc062c2f2-maas-api-service-ca\") pod \"b88da0c0-5e7b-42a1-9742-f88cc062c2f2\" (UID: \"b88da0c0-5e7b-42a1-9742-f88cc062c2f2\") "
Oct 30 20:00:35.789595 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:35.789565    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88da0c0-5e7b-42a1-9742-f88cc062c2f2-kube-api-access-zlgk5\" (UniqueName: \"kubernetes.io/projected/b88da0c0-5e7b-42a1-9742-f88cc062c2f2-kube-api-access-zlgk5\") pod \"b88da0c0-5e7b-42a1-9742-f88cc062c2f2\" (UID: \"b88da0c0-5e7b-42a1-9742-f88cc062c2f2\") "
Oct 30 20:00:35.789779 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:35.789755    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88da0c0-5e7b-42a1-9742-f88cc062c2f2-maas-api-service-ca" pod "b88da0c0-5e7b-42a1-9742-f88cc062c2f2" (UID: "b88da0c0-5e7b-42a1-9742-f88cc062c2f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:35.790635 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:35.790615    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88da0c0-5e7b-42a1-9742-f88cc062c2f2-maas-api-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:35.791732 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:35.791708    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88da0c0-5e7b-42a1-9742-f88cc062c2f2-kube-api-access-zlgk5" pod "b88da0c0-5e7b-42a1-9742-f88cc062c2f2" (UID: "b88da0c0-5e7b-42a1-9742-f88cc062c2f2"). InnerVolumeSpecName "kube-api-access-zlgk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:35.892751 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:35.892684    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlgk5\" (UniqueName: \"kubernetes.io/projected/b88da0c0-5e7b-42a1-9742-f88cc062c2f2-kube-api-access-zlgk5\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:36.189776 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.189645    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb" podUID="f94a598d-85ad-41bc-afca-a4e45dcf6f06" containerName="payload-processing" containerID="cri-o://7195c2ce753542ab552325341190e900a77afe86d65e402b10c170f56d4ea296" gracePeriod=25
Oct 30 20:00:36.206326 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.206279    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c" podUID="2375fc63-7566-4334-98a8-7436a6ddb6ae" containerName="payload-pre-processing" containerID="cri-o://9e3b0505ad64f63fe1e34b3d0278247d9d771c0a38a18c69705c7986a0211eee" gracePeriod=25
Oct 30 20:00:36.376377 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.376093    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8" event={"ID":"b88da0c0-5e7b-42a1-9742-f88cc062c2f2","Type":"ContainerDied","Data":"2356bb53e6268792fd24cb7e84f673650c8ae3b61cee2eedd3d2dc92b6b259ac"}
Oct 30 20:00:36.376377 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.376167    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8"
Oct 30 20:00:36.376555 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.376532    5408 scope.go:122] "RemoveContainer" containerID="2359d4d3a7799bbacf8490dbbdb6c347f28a134809b0be95af1ecc3a7938060e"
Oct 30 20:00:36.380175 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.379803    5408 generic.go:356] "Generic (PLEG): container finished" podID="2375fc63-7566-4334-98a8-7436a6ddb6ae" containerID="9e3b0505ad64f63fe1e34b3d0278247d9d771c0a38a18c69705c7986a0211eee" exitCode=0
Oct 30 20:00:36.380175 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.379877    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c" event={"ID":"2375fc63-7566-4334-98a8-7436a6ddb6ae","Type":"ContainerDied","Data":"9e3b0505ad64f63fe1e34b3d0278247d9d771c0a38a18c69705c7986a0211eee"}
Oct 30 20:00:36.383110 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.382925    5408 generic.go:356] "Generic (PLEG): container finished" podID="f94a598d-85ad-41bc-afca-a4e45dcf6f06" containerID="7195c2ce753542ab552325341190e900a77afe86d65e402b10c170f56d4ea296" exitCode=0
Oct 30 20:00:36.383110 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.383022    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb" event={"ID":"f94a598d-85ad-41bc-afca-a4e45dcf6f06","Type":"ContainerDied","Data":"7195c2ce753542ab552325341190e900a77afe86d65e402b10c170f56d4ea296"}
Oct 30 20:00:36.499907 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.499866    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8"]
Oct 30 20:00:36.531020 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.530993    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889840-mf2d8"]
Oct 30 20:00:36.577962 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.577931    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb"
Oct 30 20:00:36.645610 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.645561    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f94a598d-85ad-41bc-afca-a4e45dcf6f06-kube-api-access-d4nfw\" (UniqueName: \"kubernetes.io/projected/f94a598d-85ad-41bc-afca-a4e45dcf6f06-kube-api-access-d4nfw\") pod \"f94a598d-85ad-41bc-afca-a4e45dcf6f06\" (UID: \"f94a598d-85ad-41bc-afca-a4e45dcf6f06\") "
Oct 30 20:00:36.645610 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.645602    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f94a598d-85ad-41bc-afca-a4e45dcf6f06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94a598d-85ad-41bc-afca-a4e45dcf6f06-plugins-config-volume\") pod \"f94a598d-85ad-41bc-afca-a4e45dcf6f06\" (UID: \"f94a598d-85ad-41bc-afca-a4e45dcf6f06\") "
Oct 30 20:00:36.646285 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.646244    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f94a598d-85ad-41bc-afca-a4e45dcf6f06-plugins-config-volume" pod "f94a598d-85ad-41bc-afca-a4e45dcf6f06" (UID: "f94a598d-85ad-41bc-afca-a4e45dcf6f06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:36.651457 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.651432    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94a598d-85ad-41bc-afca-a4e45dcf6f06-kube-api-access-d4nfw" pod "f94a598d-85ad-41bc-afca-a4e45dcf6f06" (UID: "f94a598d-85ad-41bc-afca-a4e45dcf6f06"). InnerVolumeSpecName "kube-api-access-d4nfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:36.711479 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.711424    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c"
Oct 30 20:00:36.746938 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.746916    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2375fc63-7566-4334-98a8-7436a6ddb6ae-kube-api-access-h7fv4\" (UniqueName: \"kubernetes.io/projected/2375fc63-7566-4334-98a8-7436a6ddb6ae-kube-api-access-h7fv4\") pod \"2375fc63-7566-4334-98a8-7436a6ddb6ae\" (UID: \"2375fc63-7566-4334-98a8-7436a6ddb6ae\") "
Oct 30 20:00:36.747109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.747021    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2375fc63-7566-4334-98a8-7436a6ddb6ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2375fc63-7566-4334-98a8-7436a6ddb6ae-plugins-config-volume\") pod \"2375fc63-7566-4334-98a8-7436a6ddb6ae\" (UID: \"2375fc63-7566-4334-98a8-7436a6ddb6ae\") "
Oct 30 20:00:36.747442 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.747416    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2375fc63-7566-4334-98a8-7436a6ddb6ae-plugins-config-volume" pod "2375fc63-7566-4334-98a8-7436a6ddb6ae" (UID: "2375fc63-7566-4334-98a8-7436a6ddb6ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:36.747842 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.747822    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2375fc63-7566-4334-98a8-7436a6ddb6ae-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:36.747904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.747847    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4nfw\" (UniqueName: \"kubernetes.io/projected/f94a598d-85ad-41bc-afca-a4e45dcf6f06-kube-api-access-d4nfw\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:36.747904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.747862    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94a598d-85ad-41bc-afca-a4e45dcf6f06-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:36.749114 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.749083    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2375fc63-7566-4334-98a8-7436a6ddb6ae-kube-api-access-h7fv4" pod "2375fc63-7566-4334-98a8-7436a6ddb6ae" (UID: "2375fc63-7566-4334-98a8-7436a6ddb6ae"). InnerVolumeSpecName "kube-api-access-h7fv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:36.849435 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:36.849401    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7fv4\" (UniqueName: \"kubernetes.io/projected/2375fc63-7566-4334-98a8-7436a6ddb6ae-kube-api-access-h7fv4\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:37.439051 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:37.433909    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c" event={"ID":"2375fc63-7566-4334-98a8-7436a6ddb6ae","Type":"ContainerDied","Data":"3e9125319ab423307aeb3efa94a2318c1fffc1272947cfceb5c652f6bd5068f1"}
Oct 30 20:00:37.439051 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:37.433987    5408 scope.go:122] "RemoveContainer" containerID="9e3b0505ad64f63fe1e34b3d0278247d9d771c0a38a18c69705c7986a0211eee"
Oct 30 20:00:37.439051 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:37.434121    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c"
Oct 30 20:00:37.442211 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:37.442182    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb" event={"ID":"f94a598d-85ad-41bc-afca-a4e45dcf6f06","Type":"ContainerDied","Data":"6de6ba4e995c1d24469f6d7bdfb843805164d95e972ba54ebf6ef63f58049c47"}
Oct 30 20:00:37.442308 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:37.442296    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb"
Oct 30 20:00:37.454303 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:37.454285    5408 scope.go:122] "RemoveContainer" containerID="7195c2ce753542ab552325341190e900a77afe86d65e402b10c170f56d4ea296"
Oct 30 20:00:37.504837 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:37.504800    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb"]
Oct 30 20:00:37.534881 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:37.534850    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b88da0c0-5e7b-42a1-9742-f88cc062c2f2" path="/var/lib/kubelet/pods/b88da0c0-5e7b-42a1-9742-f88cc062c2f2/volumes"
Oct 30 20:00:37.535264 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:37.535246    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eaa5cd-75b4bc76f6-vfnnb"]
Oct 30 20:00:37.576547 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:37.576516    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c"]
Oct 30 20:00:37.594786 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:37.594743    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c"]
Oct 30 20:00:37.638760 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:37.638710    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eaa5cd-6f97c87547-4jb7c" podUID="2375fc63-7566-4334-98a8-7436a6ddb6ae" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.102:50052\" within 1s: context deadline exceeded"
Oct 30 20:00:37.638917 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:00:37.638773    5408 logging.go:55] [core] [Channel #860 SubChannel #861]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.102:50052", ServerName: "10.134.0.102:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.102:50052: operation was canceled"
Oct 30 20:00:39.597056 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:39.582085    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2375fc63-7566-4334-98a8-7436a6ddb6ae" path="/var/lib/kubelet/pods/2375fc63-7566-4334-98a8-7436a6ddb6ae/volumes"
Oct 30 20:00:39.597056 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:39.583125    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f94a598d-85ad-41bc-afca-a4e45dcf6f06" path="/var/lib/kubelet/pods/f94a598d-85ad-41bc-afca-a4e45dcf6f06/volumes"
Oct 30 20:00:39.661403 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:39.661370    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:00:41.719021 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:41.718962    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"]
Oct 30 20:00:41.719860 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:41.719825    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m" podUID="4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" containerName="istio-proxy" containerID="cri-o://4f18ddb603efa9d64eb0330288224bb0aafd48a286ac36a0cb63e991892981c5" gracePeriod=30
Oct 30 20:00:44.371355 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.371323    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"]
Oct 30 20:00:44.372279 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372256    5408 cpu_manager.go:414] "containerMap: removing container" podUID="29d94ec1-c6ac-42ad-8244-2df9bf785f01" containerName="cleanup"
Oct 30 20:00:44.372279 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372278    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d94ec1-c6ac-42ad-8244-2df9bf785f01" containerName="cleanup"
Oct 30 20:00:44.372459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372291    5408 cpu_manager.go:414] "containerMap: removing container" podUID="53f59233-5357-4386-9f05-2acfa34200dd" containerName="cleanup"
Oct 30 20:00:44.372459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372299    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f59233-5357-4386-9f05-2acfa34200dd" containerName="cleanup"
Oct 30 20:00:44.372459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372313    5408 cpu_manager.go:414] "containerMap: removing container" podUID="da1e7396-fe2d-4cb7-a019-0f6323f3b151" containerName="cleanup"
Oct 30 20:00:44.372459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372321    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da1e7396-fe2d-4cb7-a019-0f6323f3b151" containerName="cleanup"
Oct 30 20:00:44.372459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372331    5408 cpu_manager.go:414] "containerMap: removing container" podUID="21b60350-a468-4dec-afb8-c1f67c3939e5" containerName="cleanup"
Oct 30 20:00:44.372459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372338    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b60350-a468-4dec-afb8-c1f67c3939e5" containerName="cleanup"
Oct 30 20:00:44.372459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372349    5408 cpu_manager.go:414] "containerMap: removing container" podUID="29d94ec1-c6ac-42ad-8244-2df9bf785f01" containerName="cleanup"
Oct 30 20:00:44.372459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372356    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d94ec1-c6ac-42ad-8244-2df9bf785f01" containerName="cleanup"
Oct 30 20:00:44.372459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372371    5408 cpu_manager.go:414] "containerMap: removing container" podUID="39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03" containerName="cleanup"
Oct 30 20:00:44.372459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372379    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03" containerName="cleanup"
Oct 30 20:00:44.372459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372388    5408 cpu_manager.go:414] "containerMap: removing container" podUID="21b60350-a468-4dec-afb8-c1f67c3939e5" containerName="cleanup"
Oct 30 20:00:44.372459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372394    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b60350-a468-4dec-afb8-c1f67c3939e5" containerName="cleanup"
Oct 30 20:00:44.372459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372412    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f940d6ee-3ee6-4484-8689-5ece47e6c1ab" containerName="cleanup"
Oct 30 20:00:44.372459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372419    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f940d6ee-3ee6-4484-8689-5ece47e6c1ab" containerName="cleanup"
Oct 30 20:00:44.372459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372429    5408 cpu_manager.go:414] "containerMap: removing container" podUID="3a90bbcb-2fbc-4058-a5f0-3b78bde798bc" containerName="cleanup"
Oct 30 20:00:44.372459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372436    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a90bbcb-2fbc-4058-a5f0-3b78bde798bc" containerName="cleanup"
Oct 30 20:00:44.372459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372450    5408 cpu_manager.go:414] "containerMap: removing container" podUID="943fe46f-f061-4c37-80e8-0958e7e145c3" containerName="cleanup"
Oct 30 20:00:44.372459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372456    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943fe46f-f061-4c37-80e8-0958e7e145c3" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372469    5408 cpu_manager.go:414] "containerMap: removing container" podUID="3a90bbcb-2fbc-4058-a5f0-3b78bde798bc" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372476    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a90bbcb-2fbc-4058-a5f0-3b78bde798bc" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372490    5408 cpu_manager.go:414] "containerMap: removing container" podUID="da1e7396-fe2d-4cb7-a019-0f6323f3b151" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372497    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da1e7396-fe2d-4cb7-a019-0f6323f3b151" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372508    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f94a598d-85ad-41bc-afca-a4e45dcf6f06" containerName="payload-processing"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372514    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f94a598d-85ad-41bc-afca-a4e45dcf6f06" containerName="payload-processing"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372524    5408 cpu_manager.go:414] "containerMap: removing container" podUID="53f59233-5357-4386-9f05-2acfa34200dd" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372530    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f59233-5357-4386-9f05-2acfa34200dd" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372540    5408 cpu_manager.go:414] "containerMap: removing container" podUID="39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372546    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372558    5408 cpu_manager.go:414] "containerMap: removing container" podUID="8212a072-9983-47fd-9f83-08cd1a34747c" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372564    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8212a072-9983-47fd-9f83-08cd1a34747c" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372598    5408 cpu_manager.go:414] "containerMap: removing container" podUID="b88da0c0-5e7b-42a1-9742-f88cc062c2f2" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372605    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88da0c0-5e7b-42a1-9742-f88cc062c2f2" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372660    5408 cpu_manager.go:414] "containerMap: removing container" podUID="ac08d47c-3b2b-4825-8f49-c8283b06ad63" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372669    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac08d47c-3b2b-4825-8f49-c8283b06ad63" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372682    5408 cpu_manager.go:414] "containerMap: removing container" podUID="8212a072-9983-47fd-9f83-08cd1a34747c" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372689    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8212a072-9983-47fd-9f83-08cd1a34747c" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372700    5408 cpu_manager.go:414] "containerMap: removing container" podUID="b88da0c0-5e7b-42a1-9742-f88cc062c2f2" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372707    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88da0c0-5e7b-42a1-9742-f88cc062c2f2" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372718    5408 cpu_manager.go:414] "containerMap: removing container" podUID="ac08d47c-3b2b-4825-8f49-c8283b06ad63" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372725    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac08d47c-3b2b-4825-8f49-c8283b06ad63" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372744    5408 cpu_manager.go:414] "containerMap: removing container" podUID="2375fc63-7566-4334-98a8-7436a6ddb6ae" containerName="payload-pre-processing"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372752    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2375fc63-7566-4334-98a8-7436a6ddb6ae" containerName="payload-pre-processing"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372761    5408 cpu_manager.go:414] "containerMap: removing container" podUID="3a90bbcb-2fbc-4058-a5f0-3b78bde798bc" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372769    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a90bbcb-2fbc-4058-a5f0-3b78bde798bc" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372791    5408 cpu_manager.go:414] "containerMap: removing container" podUID="943fe46f-f061-4c37-80e8-0958e7e145c3" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372831    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943fe46f-f061-4c37-80e8-0958e7e145c3" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372845    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f940d6ee-3ee6-4484-8689-5ece47e6c1ab" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372852    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f940d6ee-3ee6-4484-8689-5ece47e6c1ab" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372868    5408 cpu_manager.go:414] "containerMap: removing container" podUID="943fe46f-f061-4c37-80e8-0958e7e145c3" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.372875    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943fe46f-f061-4c37-80e8-0958e7e145c3" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373078    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="2375fc63-7566-4334-98a8-7436a6ddb6ae" containerName="payload-pre-processing"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373090    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a90bbcb-2fbc-4058-a5f0-3b78bde798bc" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373103    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d94ec1-c6ac-42ad-8244-2df9bf785f01" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373114    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="f940d6ee-3ee6-4484-8689-5ece47e6c1ab" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373123    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88da0c0-5e7b-42a1-9742-f88cc062c2f2" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373131    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f59233-5357-4386-9f05-2acfa34200dd" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373138    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373146    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac08d47c-3b2b-4825-8f49-c8283b06ad63" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373154    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="943fe46f-f061-4c37-80e8-0958e7e145c3" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373164    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="f940d6ee-3ee6-4484-8689-5ece47e6c1ab" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373173    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="8212a072-9983-47fd-9f83-08cd1a34747c" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373182    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="943fe46f-f061-4c37-80e8-0958e7e145c3" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373192    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d94ec1-c6ac-42ad-8244-2df9bf785f01" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373202    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f59233-5357-4386-9f05-2acfa34200dd" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373212    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="da1e7396-fe2d-4cb7-a019-0f6323f3b151" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373219    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b60350-a468-4dec-afb8-c1f67c3939e5" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373228    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d94ec1-c6ac-42ad-8244-2df9bf785f01" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373237    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b60350-a468-4dec-afb8-c1f67c3939e5" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373253    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a90bbcb-2fbc-4058-a5f0-3b78bde798bc" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373262    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="943fe46f-f061-4c37-80e8-0958e7e145c3" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373274    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="da1e7396-fe2d-4cb7-a019-0f6323f3b151" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373284    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="f94a598d-85ad-41bc-afca-a4e45dcf6f06" containerName="payload-processing"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373294    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f59233-5357-4386-9f05-2acfa34200dd" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373303    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373318    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac08d47c-3b2b-4825-8f49-c8283b06ad63" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373329    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="8212a072-9983-47fd-9f83-08cd1a34747c" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373339    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88da0c0-5e7b-42a1-9742-f88cc062c2f2" containerName="cleanup"
Oct 30 20:00:44.373493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.373348    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac08d47c-3b2b-4825-8f49-c8283b06ad63" containerName="cleanup"
Oct 30 20:00:44.379645 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.379615    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"
Oct 30 20:00:44.389225 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.389199    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"]
Oct 30 20:00:44.436069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.432385    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c50e1e-serving-cert\""
Oct 30 20:00:44.566057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.565177    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02181f8c-536e-4aa8-bc50-049a4cf03b45-maas-api-tls\") pod \"maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f\" (UID: \"02181f8c-536e-4aa8-bc50-049a4cf03b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"
Oct 30 20:00:44.566057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.565315    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kgb4\" (UniqueName: \"kubernetes.io/projected/02181f8c-536e-4aa8-bc50-049a4cf03b45-kube-api-access-8kgb4\") pod \"maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f\" (UID: \"02181f8c-536e-4aa8-bc50-049a4cf03b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"
Oct 30 20:00:44.566057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.565421    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02181f8c-536e-4aa8-bc50-049a4cf03b45-openshift-service-ca\") pod \"maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f\" (UID: \"02181f8c-536e-4aa8-bc50-049a4cf03b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"
Oct 30 20:00:44.566057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.565445    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02181f8c-536e-4aa8-bc50-049a4cf03b45-metrics-certs\") pod \"maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f\" (UID: \"02181f8c-536e-4aa8-bc50-049a4cf03b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"
Oct 30 20:00:44.679167 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.672184    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02181f8c-536e-4aa8-bc50-049a4cf03b45-maas-api-tls\") pod \"maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f\" (UID: \"02181f8c-536e-4aa8-bc50-049a4cf03b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"
Oct 30 20:00:44.679167 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.672369    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kgb4\" (UniqueName: \"kubernetes.io/projected/02181f8c-536e-4aa8-bc50-049a4cf03b45-kube-api-access-8kgb4\") pod \"maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f\" (UID: \"02181f8c-536e-4aa8-bc50-049a4cf03b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"
Oct 30 20:00:44.679167 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.672491    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02181f8c-536e-4aa8-bc50-049a4cf03b45-openshift-service-ca\") pod \"maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f\" (UID: \"02181f8c-536e-4aa8-bc50-049a4cf03b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"
Oct 30 20:00:44.679167 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.672521    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02181f8c-536e-4aa8-bc50-049a4cf03b45-metrics-certs\") pod \"maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f\" (UID: \"02181f8c-536e-4aa8-bc50-049a4cf03b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"
Oct 30 20:00:44.679167 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.674510    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02181f8c-536e-4aa8-bc50-049a4cf03b45-openshift-service-ca\") pod \"maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f\" (UID: \"02181f8c-536e-4aa8-bc50-049a4cf03b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"
Oct 30 20:00:44.696797 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.696769    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kgb4\" (UniqueName: \"kubernetes.io/projected/02181f8c-536e-4aa8-bc50-049a4cf03b45-kube-api-access-8kgb4\") pod \"maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f\" (UID: \"02181f8c-536e-4aa8-bc50-049a4cf03b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"
Oct 30 20:00:44.699562 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.699519    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02181f8c-536e-4aa8-bc50-049a4cf03b45-metrics-certs\") pod \"maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f\" (UID: \"02181f8c-536e-4aa8-bc50-049a4cf03b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"
Oct 30 20:00:44.727111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.699974    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02181f8c-536e-4aa8-bc50-049a4cf03b45-maas-api-tls\") pod \"maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f\" (UID: \"02181f8c-536e-4aa8-bc50-049a4cf03b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"
Oct 30 20:00:44.727111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:44.701387    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"
Oct 30 20:00:45.234285 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:45.219340    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"]
Oct 30 20:00:45.802236 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:45.802198    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f" event={"ID":"02181f8c-536e-4aa8-bc50-049a4cf03b45","Type":"ContainerStarted","Data":"dd7d17d301fb0b710a24ae323570371b7f10b1663d49c0fb867a88a633e65397"}
Oct 30 20:00:46.836103 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:46.836066    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f" event={"ID":"02181f8c-536e-4aa8-bc50-049a4cf03b45","Type":"ContainerStarted","Data":"96f536e70186253e7c3c864e0740a6ac29114338c9139105249c47c1b20923ba"}
Oct 30 20:00:46.836511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:46.836124    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"
Oct 30 20:00:46.859341 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:46.859295    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f" podStartSLOduration=2.290320089 podStartE2EDuration="2.859282186s" totalImagesPullingTime="568.962097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:46.857661961 +0000 UTC m=+1801.441931605" watchObservedRunningTime="2026-10-30 20:00:46.859282186 +0000 UTC m=+1801.443551802"
Oct 30 20:00:47.429530 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.429494    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 20:00:47.560459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.560423    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-envoy\") pod \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") "
Oct 30 20:00:47.560619 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.560497    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-workload-certs\") pod \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") "
Oct 30 20:00:47.560619 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.560567    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-credential-socket\") pod \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") "
Oct 30 20:00:47.560619 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.560611    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-kube-api-access-2dwh4\" (UniqueName: \"kubernetes.io/projected/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-kube-api-access-2dwh4\") pod \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") "
Oct 30 20:00:47.560786 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.560646    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-workload-socket\") pod \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") "
Oct 30 20:00:47.560786 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.560701    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istiod-ca-cert\") pod \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") "
Oct 30 20:00:47.560899 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.560827    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-workload-certs" pod "4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" (UID: "4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:47.560971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.560945    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-credential-socket" pod "4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" (UID: "4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:47.560971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.560964    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-token\" (UniqueName: \"kubernetes.io/projected/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-token\") pod \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") "
Oct 30 20:00:47.561136 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.560965    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-workload-socket" pod "4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" (UID: "4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:47.561136 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.561024    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-podinfo\") pod \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") "
Oct 30 20:00:47.561136 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.561107    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-data\") pod \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\" (UID: \"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7\") "
Oct 30 20:00:47.561312 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.561214    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istiod-ca-cert" pod "4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" (UID: "4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:47.561575 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.561548    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-data" pod "4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" (UID: "4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:47.562065 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.562011    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.562161 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.562072    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.562161 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.562089    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.562161 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.562102    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.562161 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.562115    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.564312 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.564287    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-token" pod "4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" (UID: "4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:47.564548 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.564524    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-podinfo" pod "4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" (UID: "4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:00:47.564637 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.564559    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-envoy" pod "4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" (UID: "4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:47.566311 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.566284    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-kube-api-access-2dwh4" pod "4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" (UID: "4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7"). InnerVolumeSpecName "kube-api-access-2dwh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:47.663494 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.663414    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.663494 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.663448    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.663494 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.663458    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.663494 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.663467    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dwh4\" (UniqueName: \"kubernetes.io/projected/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7-kube-api-access-2dwh4\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.845272 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.845228    5408 generic.go:356] "Generic (PLEG): container finished" podID="4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" containerID="4f18ddb603efa9d64eb0330288224bb0aafd48a286ac36a0cb63e991892981c5" exitCode=0
Oct 30 20:00:47.845772 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.845289    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"
Oct 30 20:00:47.845772 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.845314    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m" event={"ID":"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7","Type":"ContainerDied","Data":"4f18ddb603efa9d64eb0330288224bb0aafd48a286ac36a0cb63e991892981c5"}
Oct 30 20:00:47.845772 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.845376    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m" event={"ID":"4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7","Type":"ContainerDied","Data":"821ac2d65e3f9cf8ffed35bb99876e43b1ccca20fbdafc72c9c641627f4cc558"}
Oct 30 20:00:47.845772 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.845407    5408 scope.go:122] "RemoveContainer" containerID="4f18ddb603efa9d64eb0330288224bb0aafd48a286ac36a0cb63e991892981c5"
Oct 30 20:00:47.858288 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.858269    5408 scope.go:122] "RemoveContainer" containerID="4f18ddb603efa9d64eb0330288224bb0aafd48a286ac36a0cb63e991892981c5"
Oct 30 20:00:47.858597 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:00:47.858578    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f18ddb603efa9d64eb0330288224bb0aafd48a286ac36a0cb63e991892981c5\": container with ID starting with 4f18ddb603efa9d64eb0330288224bb0aafd48a286ac36a0cb63e991892981c5 not found: ID does not exist" containerID="4f18ddb603efa9d64eb0330288224bb0aafd48a286ac36a0cb63e991892981c5"
Oct 30 20:00:47.858652 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.858606    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f18ddb603efa9d64eb0330288224bb0aafd48a286ac36a0cb63e991892981c5"} err="failed to get container status \"4f18ddb603efa9d64eb0330288224bb0aafd48a286ac36a0cb63e991892981c5\": rpc error: code = NotFound desc = could not find container \"4f18ddb603efa9d64eb0330288224bb0aafd48a286ac36a0cb63e991892981c5\": container with ID starting with 4f18ddb603efa9d64eb0330288224bb0aafd48a286ac36a0cb63e991892981c5 not found: ID does not exist"
Oct 30 20:00:47.872984 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.872959    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"]
Oct 30 20:00:47.878655 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:47.878621    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-4rv4m"]
Oct 30 20:00:49.534811 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:49.534771    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" path="/var/lib/kubelet/pods/4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7/volumes"
Oct 30 20:00:52.853767 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:52.853726    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"
Oct 30 20:00:53.955208 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:53.955130    5408 generic.go:356] "Generic (PLEG): container finished" podID="a6aa0a29-6f37-4b42-92f1-95707639c58a" containerID="5a96b2d2ea39ce225c4f3585e4688c75559180b7d079cad0938a4fb0bffcf7d6" exitCode=0
Oct 30 20:00:53.955208 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:53.955173    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f" event={"ID":"a6aa0a29-6f37-4b42-92f1-95707639c58a","Type":"ContainerDied","Data":"5a96b2d2ea39ce225c4f3585e4688c75559180b7d079cad0938a4fb0bffcf7d6"}
Oct 30 20:00:53.955675 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:53.955660    5408 scope.go:122] "RemoveContainer" containerID="5a96b2d2ea39ce225c4f3585e4688c75559180b7d079cad0938a4fb0bffcf7d6"
Oct 30 20:00:53.957722 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:53.957692    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:00:53.957804 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:53.957769    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:00:53.957841 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:53.957803    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:00:54.849270 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:54.849227    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 20:00:54.964547 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:54.964514    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f" event={"ID":"a6aa0a29-6f37-4b42-92f1-95707639c58a","Type":"ContainerStarted","Data":"d7ff1041f51b112c2c23effd347b3d6e35fb6eb39c177d18f104301c0939e91a"}
Oct 30 20:00:55.159938 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:55.159852    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"]
Oct 30 20:00:55.160167 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:55.160112    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4" podUID="890daa8d-efaa-423f-8d7c-172b4e6081ff" containerName="maas-api" containerID="cri-o://0f8b945db3af662cb17ff892767f91465945d4c5193c9dd40434337093d8e05d" gracePeriod=30
Oct 30 20:00:55.295319 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:55.295294    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 20:00:55.296536 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:55.296512    5408 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 20:00:55.296642 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:55.296556    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f" podUID="a6aa0a29-6f37-4b42-92f1-95707639c58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 20:00:56.041144 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.034774    5408 generic.go:356] "Generic (PLEG): container finished" podID="890daa8d-efaa-423f-8d7c-172b4e6081ff" containerID="0f8b945db3af662cb17ff892767f91465945d4c5193c9dd40434337093d8e05d" exitCode=0
Oct 30 20:00:56.041144 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.035986    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4" event={"ID":"890daa8d-efaa-423f-8d7c-172b4e6081ff","Type":"ContainerDied","Data":"0f8b945db3af662cb17ff892767f91465945d4c5193c9dd40434337093d8e05d"}
Oct 30 20:00:56.126065 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.111561    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 20:00:56.242850 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.242818    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/890daa8d-efaa-423f-8d7c-172b4e6081ff-kube-api-access-74ph9\" (UniqueName: \"kubernetes.io/projected/890daa8d-efaa-423f-8d7c-172b4e6081ff-kube-api-access-74ph9\") pod \"890daa8d-efaa-423f-8d7c-172b4e6081ff\" (UID: \"890daa8d-efaa-423f-8d7c-172b4e6081ff\") "
Oct 30 20:00:56.243008 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.242956    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/890daa8d-efaa-423f-8d7c-172b4e6081ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/890daa8d-efaa-423f-8d7c-172b4e6081ff-metrics-certs\") pod \"890daa8d-efaa-423f-8d7c-172b4e6081ff\" (UID: \"890daa8d-efaa-423f-8d7c-172b4e6081ff\") "
Oct 30 20:00:56.243008 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.242994    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/890daa8d-efaa-423f-8d7c-172b4e6081ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/890daa8d-efaa-423f-8d7c-172b4e6081ff-openshift-service-ca\") pod \"890daa8d-efaa-423f-8d7c-172b4e6081ff\" (UID: \"890daa8d-efaa-423f-8d7c-172b4e6081ff\") "
Oct 30 20:00:56.243247 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.243226    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/890daa8d-efaa-423f-8d7c-172b4e6081ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/890daa8d-efaa-423f-8d7c-172b4e6081ff-maas-api-tls\") pod \"890daa8d-efaa-423f-8d7c-172b4e6081ff\" (UID: \"890daa8d-efaa-423f-8d7c-172b4e6081ff\") "
Oct 30 20:00:56.243441 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.243416    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/890daa8d-efaa-423f-8d7c-172b4e6081ff-openshift-service-ca" pod "890daa8d-efaa-423f-8d7c-172b4e6081ff" (UID: "890daa8d-efaa-423f-8d7c-172b4e6081ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:56.244063 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.244023    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/890daa8d-efaa-423f-8d7c-172b4e6081ff-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:56.245064 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.245008    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/890daa8d-efaa-423f-8d7c-172b4e6081ff-metrics-certs" pod "890daa8d-efaa-423f-8d7c-172b4e6081ff" (UID: "890daa8d-efaa-423f-8d7c-172b4e6081ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:56.245309 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.245292    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/890daa8d-efaa-423f-8d7c-172b4e6081ff-kube-api-access-74ph9" pod "890daa8d-efaa-423f-8d7c-172b4e6081ff" (UID: "890daa8d-efaa-423f-8d7c-172b4e6081ff"). InnerVolumeSpecName "kube-api-access-74ph9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:56.245381 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.245360    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/890daa8d-efaa-423f-8d7c-172b4e6081ff-maas-api-tls" pod "890daa8d-efaa-423f-8d7c-172b4e6081ff" (UID: "890daa8d-efaa-423f-8d7c-172b4e6081ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:56.295291 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.295265    5408 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 20:00:56.295399 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.295325    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f" podUID="a6aa0a29-6f37-4b42-92f1-95707639c58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 20:00:56.354089 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.352565    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/890daa8d-efaa-423f-8d7c-172b4e6081ff-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:56.354089 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.352604    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/890daa8d-efaa-423f-8d7c-172b4e6081ff-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:56.354089 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.352620    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74ph9\" (UniqueName: \"kubernetes.io/projected/890daa8d-efaa-423f-8d7c-172b4e6081ff-kube-api-access-74ph9\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:56.743818 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.731219    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"]
Oct 30 20:00:56.743818 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:56.731461    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc" podUID="084af46f-59a7-4518-8938-2b4e09979c5f" containerName="istio-proxy" containerID="cri-o://3ce5c6d61ebeb949b8567f999d0df77009af293288151ad4b7fa388da8f8cb91" gracePeriod=30
Oct 30 20:00:57.078150 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.075872    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"
Oct 30 20:00:57.078150 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.077860    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4" event={"ID":"890daa8d-efaa-423f-8d7c-172b4e6081ff","Type":"ContainerDied","Data":"16d18f4e7139ddf5e92a4e92898f9cd784a3c3dcca9b86362e670dd8457020f3"}
Oct 30 20:00:57.078150 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.077922    5408 scope.go:122] "RemoveContainer" containerID="0f8b945db3af662cb17ff892767f91465945d4c5193c9dd40434337093d8e05d"
Oct 30 20:00:57.270715 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.270683    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"]
Oct 30 20:00:57.303174 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.303129    5408 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 20:00:57.303432 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.303401    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f" podUID="a6aa0a29-6f37-4b42-92f1-95707639c58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 20:00:57.306741 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.306712    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eaa5cd-6597f7b974-m52v4"]
Oct 30 20:00:57.447215 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.447138    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"]
Oct 30 20:00:57.452100 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.452074    5408 cpu_manager.go:414] "containerMap: removing container" podUID="da1e7396-fe2d-4cb7-a019-0f6323f3b151" containerName="cleanup"
Oct 30 20:00:57.452266 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.452251    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da1e7396-fe2d-4cb7-a019-0f6323f3b151" containerName="cleanup"
Oct 30 20:00:57.452378 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.452366    5408 cpu_manager.go:414] "containerMap: removing container" podUID="53f59233-5357-4386-9f05-2acfa34200dd" containerName="cleanup"
Oct 30 20:00:57.452464 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.452452    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f59233-5357-4386-9f05-2acfa34200dd" containerName="cleanup"
Oct 30 20:00:57.452566 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.452556    5408 cpu_manager.go:414] "containerMap: removing container" podUID="8212a072-9983-47fd-9f83-08cd1a34747c" containerName="cleanup"
Oct 30 20:00:57.452648 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.452637    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8212a072-9983-47fd-9f83-08cd1a34747c" containerName="cleanup"
Oct 30 20:00:57.465491 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.452774    5408 cpu_manager.go:414] "containerMap: removing container" podUID="4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" containerName="istio-proxy"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.465647    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" containerName="istio-proxy"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.472392    5408 cpu_manager.go:414] "containerMap: removing container" podUID="b88da0c0-5e7b-42a1-9742-f88cc062c2f2" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.472405    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88da0c0-5e7b-42a1-9742-f88cc062c2f2" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.472427    5408 cpu_manager.go:414] "containerMap: removing container" podUID="ac08d47c-3b2b-4825-8f49-c8283b06ad63" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.472435    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac08d47c-3b2b-4825-8f49-c8283b06ad63" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.472505    5408 cpu_manager.go:414] "containerMap: removing container" podUID="890daa8d-efaa-423f-8d7c-172b4e6081ff" containerName="maas-api"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.472518    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="890daa8d-efaa-423f-8d7c-172b4e6081ff" containerName="maas-api"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.472545    5408 cpu_manager.go:414] "containerMap: removing container" podUID="29d94ec1-c6ac-42ad-8244-2df9bf785f01" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.472552    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d94ec1-c6ac-42ad-8244-2df9bf785f01" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.472568    5408 cpu_manager.go:414] "containerMap: removing container" podUID="39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.472574    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.472592    5408 cpu_manager.go:414] "containerMap: removing container" podUID="21b60350-a468-4dec-afb8-c1f67c3939e5" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.472606    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b60350-a468-4dec-afb8-c1f67c3939e5" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.472639    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f940d6ee-3ee6-4484-8689-5ece47e6c1ab" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.472646    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f940d6ee-3ee6-4484-8689-5ece47e6c1ab" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.473105    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="da1e7396-fe2d-4cb7-a019-0f6323f3b151" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.473135    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b60350-a468-4dec-afb8-c1f67c3939e5" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.473149    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a90bbcb-2fbc-4058-a5f0-3b78bde798bc" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.473170    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="39f0eaaa-aa8e-4a9b-9422-6a7dcbd37d03" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.473197    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d97d5bd-2d70-4d23-bc5f-bbcc9abf45d7" containerName="istio-proxy"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.473209    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88da0c0-5e7b-42a1-9742-f88cc062c2f2" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.473226    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="f940d6ee-3ee6-4484-8689-5ece47e6c1ab" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.473252    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="8212a072-9983-47fd-9f83-08cd1a34747c" containerName="cleanup"
Oct 30 20:00:57.478109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.473262    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="890daa8d-efaa-423f-8d7c-172b4e6081ff" containerName="maas-api"
Oct 30 20:00:57.484087 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.482292    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.484087 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.483536    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"]
Oct 30 20:00:57.495110 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.495085    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ed8b50-openshift-default-dockercfg-cx8g9\""
Oct 30 20:00:57.605690 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.598787    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-podinfo\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.605690 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.598886    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-data\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.605690 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.598946    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138e6bd8-41e4-496d-a8a0-4722f4947a80-istiod-ca-cert\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.605690 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.598998    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-credential-socket\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.605690 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.599063    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-workload-socket\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.605690 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.599170    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-workload-certs\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.605690 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.599253    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf8nl\" (UniqueName: \"kubernetes.io/projected/138e6bd8-41e4-496d-a8a0-4722f4947a80-kube-api-access-tf8nl\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.605690 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.599281    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-envoy\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.605690 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.599453    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-token\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.641055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.640620    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="890daa8d-efaa-423f-8d7c-172b4e6081ff" path="/var/lib/kubelet/pods/890daa8d-efaa-423f-8d7c-172b4e6081ff/volumes"
Oct 30 20:00:57.717763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.709221    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-workload-certs\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.717763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.709343    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf8nl\" (UniqueName: \"kubernetes.io/projected/138e6bd8-41e4-496d-a8a0-4722f4947a80-kube-api-access-tf8nl\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.717763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.709398    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-envoy\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.717763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.709517    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-token\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.717763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.709778    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-podinfo\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.717763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.709837    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-data\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.717763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.709891    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138e6bd8-41e4-496d-a8a0-4722f4947a80-istiod-ca-cert\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.717763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.709947    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-credential-socket\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.717763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.710003    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-workload-socket\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.717763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.710779    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-workload-socket\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.717763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.711188    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-workload-certs\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.717763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.712338    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-data\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.717763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.713788    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138e6bd8-41e4-496d-a8a0-4722f4947a80-istiod-ca-cert\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.725058 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.724389    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-envoy\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.730905 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.730877    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-credential-socket\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.761076 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.756723    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-podinfo\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.766276 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.763826    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf8nl\" (UniqueName: \"kubernetes.io/projected/138e6bd8-41e4-496d-a8a0-4722f4947a80-kube-api-access-tf8nl\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.798984 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.798884    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-token\") pod \"e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") " pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:57.801660 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:57.801620    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:58.295323 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:58.295281    5408 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 20:00:58.295733 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:58.295356    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f" podUID="a6aa0a29-6f37-4b42-92f1-95707639c58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 20:00:58.374422 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:58.374391    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"]
Oct 30 20:00:58.380759 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:00:58.378468    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138e6bd8_41e4_496d_a8a0_4722f4947a80.slice/crio-be4589f82bef2c48f1ce091b3d59445acecf1fa37b63d3d5be70acbca776f0e8 WatchSource:0}: Error finding container be4589f82bef2c48f1ce091b3d59445acecf1fa37b63d3d5be70acbca776f0e8: Status 404 returned error can't find the container with id be4589f82bef2c48f1ce091b3d59445acecf1fa37b63d3d5be70acbca776f0e8
Oct 30 20:00:58.381929 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:58.381895    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:00:58.382063 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:58.381969    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:00:58.382063 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:58.381999    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:00:59.158067 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:59.157229    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg" event={"ID":"138e6bd8-41e4-496d-a8a0-4722f4947a80","Type":"ContainerStarted","Data":"08a66fe5935b35dcc9aa57684afa0dcaba2b4da610edb60f9e1c8038549c5aee"}
Oct 30 20:00:59.158067 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:59.157328    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg" event={"ID":"138e6bd8-41e4-496d-a8a0-4722f4947a80","Type":"ContainerStarted","Data":"be4589f82bef2c48f1ce091b3d59445acecf1fa37b63d3d5be70acbca776f0e8"}
Oct 30 20:00:59.187848 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:59.187795    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg" podStartSLOduration=2.187781772 podStartE2EDuration="2.187781772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:59.18311692 +0000 UTC m=+1813.767386534" watchObservedRunningTime="2026-10-30 20:00:59.187781772 +0000 UTC m=+1813.772051393"
Oct 30 20:00:59.299053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:59.295693    5408 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 20:00:59.299053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:59.295748    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f" podUID="a6aa0a29-6f37-4b42-92f1-95707639c58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 20:00:59.741956 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:59.741923    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11080-ab1424"]
Oct 30 20:00:59.746569 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:59.746551    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-ab1424"
Oct 30 20:00:59.751655 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:59.751630    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xqd5g\""
Oct 30 20:00:59.765240 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:59.765221    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11080-ab1424"]
Oct 30 20:00:59.824056 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:59.802945    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klj6t\" (UniqueName: \"kubernetes.io/projected/aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3-kube-api-access-klj6t\") pod \"test-curl-11080-ab1424\" (UID: \"aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3\") " pod="opendatahub/test-curl-11080-ab1424"
Oct 30 20:00:59.828843 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:59.828812    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:59.836827 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:59.836803    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:00:59.927279 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:59.927233    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klj6t\" (UniqueName: \"kubernetes.io/projected/aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3-kube-api-access-klj6t\") pod \"test-curl-11080-ab1424\" (UID: \"aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3\") " pod="opendatahub/test-curl-11080-ab1424"
Oct 30 20:00:59.940698 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:00:59.940673    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klj6t\" (UniqueName: \"kubernetes.io/projected/aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3-kube-api-access-klj6t\") pod \"test-curl-11080-ab1424\" (UID: \"aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3\") " pod="opendatahub/test-curl-11080-ab1424"
Oct 30 20:01:00.064114 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:00.063364    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-ab1424"
Oct 30 20:01:00.165166 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:00.165137    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:01:00.166115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:00.166092    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:01:00.294960 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:00.294925    5408 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 20:01:00.295222 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:00.295004    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f" podUID="a6aa0a29-6f37-4b42-92f1-95707639c58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 20:01:00.430135 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:00.430105    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11080-ab1424"]
Oct 30 20:01:00.433455 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:01:00.433429    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa8d6bf1_0ce4_4a28_be7a_8ec300aee1b3.slice/crio-99dc1275390629a75191703f1b1e0a242c9dcbef16bdf583ae64907e24f3a079 WatchSource:0}: Error finding container 99dc1275390629a75191703f1b1e0a242c9dcbef16bdf583ae64907e24f3a079: Status 404 returned error can't find the container with id 99dc1275390629a75191703f1b1e0a242c9dcbef16bdf583ae64907e24f3a079
Oct 30 20:01:01.173270 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:01.173217    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11080-ab1424" event={"ID":"aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3","Type":"ContainerStarted","Data":"50fc0c9e67f5964ce07f7c5954ba7f2699790ce9ddd144d99ab98466305fc56c"}
Oct 30 20:01:01.173430 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:01.173278    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11080-ab1424" event={"ID":"aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3","Type":"ContainerStarted","Data":"99dc1275390629a75191703f1b1e0a242c9dcbef16bdf583ae64907e24f3a079"}
Oct 30 20:01:01.194100 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:01.194053    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11080-ab1424" podStartSLOduration=2.194039196 podStartE2EDuration="2.194039196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:01.192350472 +0000 UTC m=+1815.776620091" watchObservedRunningTime="2026-10-30 20:01:01.194039196 +0000 UTC m=+1815.778308804"
Oct 30 20:01:01.294875 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:01.294842    5408 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 20:01:01.295084 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:01.294901    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f" podUID="a6aa0a29-6f37-4b42-92f1-95707639c58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 20:01:01.993503 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:01.993467    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11080-ab1424"]
Oct 30 20:01:02.006976 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.006943    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11080-ab1424"]
Oct 30 20:01:02.110447 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.110423    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 20:01:02.183313 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.183280    5408 generic.go:356] "Generic (PLEG): container finished" podID="084af46f-59a7-4518-8938-2b4e09979c5f" containerID="3ce5c6d61ebeb949b8567f999d0df77009af293288151ad4b7fa388da8f8cb91" exitCode=0
Oct 30 20:01:02.183481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.183336    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"
Oct 30 20:01:02.183481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.183363    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc" event={"ID":"084af46f-59a7-4518-8938-2b4e09979c5f","Type":"ContainerDied","Data":"3ce5c6d61ebeb949b8567f999d0df77009af293288151ad4b7fa388da8f8cb91"}
Oct 30 20:01:02.183481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.183434    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc" event={"ID":"084af46f-59a7-4518-8938-2b4e09979c5f","Type":"ContainerDied","Data":"425b7395aee8b8049cb9e51526c5e95b111203e8ade23f8de908668a7cf3a5b5"}
Oct 30 20:01:02.183481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.183466    5408 scope.go:122] "RemoveContainer" containerID="3ce5c6d61ebeb949b8567f999d0df77009af293288151ad4b7fa388da8f8cb91"
Oct 30 20:01:02.195223 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.195197    5408 scope.go:122] "RemoveContainer" containerID="3ce5c6d61ebeb949b8567f999d0df77009af293288151ad4b7fa388da8f8cb91"
Oct 30 20:01:02.195521 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:01:02.195492    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce5c6d61ebeb949b8567f999d0df77009af293288151ad4b7fa388da8f8cb91\": container with ID starting with 3ce5c6d61ebeb949b8567f999d0df77009af293288151ad4b7fa388da8f8cb91 not found: ID does not exist" containerID="3ce5c6d61ebeb949b8567f999d0df77009af293288151ad4b7fa388da8f8cb91"
Oct 30 20:01:02.195638 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.195520    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce5c6d61ebeb949b8567f999d0df77009af293288151ad4b7fa388da8f8cb91"} err="failed to get container status \"3ce5c6d61ebeb949b8567f999d0df77009af293288151ad4b7fa388da8f8cb91\": rpc error: code = NotFound desc = could not find container \"3ce5c6d61ebeb949b8567f999d0df77009af293288151ad4b7fa388da8f8cb91\": container with ID starting with 3ce5c6d61ebeb949b8567f999d0df77009af293288151ad4b7fa388da8f8cb91 not found: ID does not exist"
Oct 30 20:01:02.212233 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.212203    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-workload-certs\") pod \"084af46f-59a7-4518-8938-2b4e09979c5f\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") "
Oct 30 20:01:02.212367 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.212278    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/084af46f-59a7-4518-8938-2b4e09979c5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084af46f-59a7-4518-8938-2b4e09979c5f-istio-podinfo\") pod \"084af46f-59a7-4518-8938-2b4e09979c5f\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") "
Oct 30 20:01:02.212436 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.212385    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-workload-socket\") pod \"084af46f-59a7-4518-8938-2b4e09979c5f\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") "
Oct 30 20:01:02.212490 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.212447    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-workload-certs" pod "084af46f-59a7-4518-8938-2b4e09979c5f" (UID: "084af46f-59a7-4518-8938-2b4e09979c5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:02.212545 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.212494    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084af46f-59a7-4518-8938-2b4e09979c5f-istio-token\" (UniqueName: \"kubernetes.io/projected/084af46f-59a7-4518-8938-2b4e09979c5f-istio-token\") pod \"084af46f-59a7-4518-8938-2b4e09979c5f\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") "
Oct 30 20:01:02.212595 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.212542    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-credential-socket\") pod \"084af46f-59a7-4518-8938-2b4e09979c5f\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") "
Oct 30 20:01:02.212595 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.212578    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-istio-data\") pod \"084af46f-59a7-4518-8938-2b4e09979c5f\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") "
Oct 30 20:01:02.212707 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.212629    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084af46f-59a7-4518-8938-2b4e09979c5f-kube-api-access-lbgx6\" (UniqueName: \"kubernetes.io/projected/084af46f-59a7-4518-8938-2b4e09979c5f-kube-api-access-lbgx6\") pod \"084af46f-59a7-4518-8938-2b4e09979c5f\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") "
Oct 30 20:01:02.212707 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.212642    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-workload-socket" pod "084af46f-59a7-4518-8938-2b4e09979c5f" (UID: "084af46f-59a7-4518-8938-2b4e09979c5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:02.212707 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.212665    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-istio-envoy\") pod \"084af46f-59a7-4518-8938-2b4e09979c5f\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") "
Oct 30 20:01:02.212854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.212818    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-credential-socket" pod "084af46f-59a7-4518-8938-2b4e09979c5f" (UID: "084af46f-59a7-4518-8938-2b4e09979c5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:02.212854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.212839    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/084af46f-59a7-4518-8938-2b4e09979c5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084af46f-59a7-4518-8938-2b4e09979c5f-istiod-ca-cert\") pod \"084af46f-59a7-4518-8938-2b4e09979c5f\" (UID: \"084af46f-59a7-4518-8938-2b4e09979c5f\") "
Oct 30 20:01:02.212959 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.212881    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-istio-data" pod "084af46f-59a7-4518-8938-2b4e09979c5f" (UID: "084af46f-59a7-4518-8938-2b4e09979c5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:02.213975 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.213286    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084af46f-59a7-4518-8938-2b4e09979c5f-istiod-ca-cert" pod "084af46f-59a7-4518-8938-2b4e09979c5f" (UID: "084af46f-59a7-4518-8938-2b4e09979c5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:02.213975 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.213769    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:02.213975 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.213837    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:02.213975 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.213853    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084af46f-59a7-4518-8938-2b4e09979c5f-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:02.213975 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.213866    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:02.213975 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.213879    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:02.214939 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.214910    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/084af46f-59a7-4518-8938-2b4e09979c5f-istio-podinfo" pod "084af46f-59a7-4518-8938-2b4e09979c5f" (UID: "084af46f-59a7-4518-8938-2b4e09979c5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:02.215081 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.215052    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084af46f-59a7-4518-8938-2b4e09979c5f-istio-token" pod "084af46f-59a7-4518-8938-2b4e09979c5f" (UID: "084af46f-59a7-4518-8938-2b4e09979c5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:02.215147 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.215089    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084af46f-59a7-4518-8938-2b4e09979c5f-kube-api-access-lbgx6" pod "084af46f-59a7-4518-8938-2b4e09979c5f" (UID: "084af46f-59a7-4518-8938-2b4e09979c5f"). InnerVolumeSpecName "kube-api-access-lbgx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:02.215278 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.215249    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-istio-envoy" pod "084af46f-59a7-4518-8938-2b4e09979c5f" (UID: "084af46f-59a7-4518-8938-2b4e09979c5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:02.300102 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.300071    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 20:01:02.303082 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.300607    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 20:01:02.303082 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.301476    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 20:01:02.315817 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.315707    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbgx6\" (UniqueName: \"kubernetes.io/projected/084af46f-59a7-4518-8938-2b4e09979c5f-kube-api-access-lbgx6\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:02.315817 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.315735    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084af46f-59a7-4518-8938-2b4e09979c5f-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:02.315817 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.315751    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084af46f-59a7-4518-8938-2b4e09979c5f-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:02.315817 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.315765    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/084af46f-59a7-4518-8938-2b4e09979c5f-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:02.516232 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.516202    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"]
Oct 30 20:01:02.520075 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:02.520049    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-zk2sc"]
Oct 30 20:01:03.134047 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.133998    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs"]
Oct 30 20:01:03.134725 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.134711    5408 cpu_manager.go:414] "containerMap: removing container" podUID="084af46f-59a7-4518-8938-2b4e09979c5f" containerName="istio-proxy"
Oct 30 20:01:03.134769 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.134726    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084af46f-59a7-4518-8938-2b4e09979c5f" containerName="istio-proxy"
Oct 30 20:01:03.134769 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.134747    5408 cpu_manager.go:414] "containerMap: removing container" podUID="aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3" containerName="test-curl-11080-ab1424"
Oct 30 20:01:03.134769 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.134754    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3" containerName="test-curl-11080-ab1424"
Oct 30 20:01:03.134880 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.134871    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="084af46f-59a7-4518-8938-2b4e09979c5f" containerName="istio-proxy"
Oct 30 20:01:03.134921 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.134886    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3" containerName="test-curl-11080-ab1424"
Oct 30 20:01:03.139025 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.138994    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs"
Oct 30 20:01:03.143326 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.143305    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2"]
Oct 30 20:01:03.147308 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.146152    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ed8b50\""
Oct 30 20:01:03.147308 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.146591    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ed8b50-dockercfg-7r4pf\""
Oct 30 20:01:03.153450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.152229    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2"
Oct 30 20:01:03.171016 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.170985    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs"]
Oct 30 20:01:03.193040 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.193004    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2"]
Oct 30 20:01:03.196425 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.196398    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11080-ab1424" podUID="aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3" containerName="test-curl-11080-ab1424" containerID="cri-o://50fc0c9e67f5964ce07f7c5954ba7f2699790ce9ddd144d99ab98466305fc56c" gracePeriod=2
Oct 30 20:01:03.214149 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:01:03.214120    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11080-ab1424\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3" pod="opendatahub/test-curl-11080-ab1424"
Oct 30 20:01:03.292057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.254346    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2\" (UID: \"ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2"
Oct 30 20:01:03.292057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.254476    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs\" (UID: \"4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs"
Oct 30 20:01:03.292057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.254528    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4sqs\" (UniqueName: \"kubernetes.io/projected/ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b-kube-api-access-k4sqs\") pod \"payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2\" (UID: \"ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2"
Oct 30 20:01:03.292057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.254809    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkjgm\" (UniqueName: \"kubernetes.io/projected/4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f-kube-api-access-rkjgm\") pod \"payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs\" (UID: \"4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs"
Oct 30 20:01:03.367951 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.367910    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2\" (UID: \"ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2"
Oct 30 20:01:03.368576 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.368542    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs\" (UID: \"4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs"
Oct 30 20:01:03.368774 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.368754    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4sqs\" (UniqueName: \"kubernetes.io/projected/ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b-kube-api-access-k4sqs\") pod \"payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2\" (UID: \"ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2"
Oct 30 20:01:03.369059 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.369015    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2\" (UID: \"ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2"
Oct 30 20:01:03.369266 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.369239    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkjgm\" (UniqueName: \"kubernetes.io/projected/4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f-kube-api-access-rkjgm\") pod \"payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs\" (UID: \"4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs"
Oct 30 20:01:03.384211 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.384141    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs\" (UID: \"4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs"
Oct 30 20:01:03.426663 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.412190    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkjgm\" (UniqueName: \"kubernetes.io/projected/4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f-kube-api-access-rkjgm\") pod \"payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs\" (UID: \"4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs"
Oct 30 20:01:03.426956 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.424563    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4sqs\" (UniqueName: \"kubernetes.io/projected/ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b-kube-api-access-k4sqs\") pod \"payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2\" (UID: \"ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2"
Oct 30 20:01:03.475788 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.475487    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs"
Oct 30 20:01:03.495359 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.495329    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2"
Oct 30 20:01:03.639214 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.637944    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="084af46f-59a7-4518-8938-2b4e09979c5f" path="/var/lib/kubelet/pods/084af46f-59a7-4518-8938-2b4e09979c5f/volumes"
Oct 30 20:01:03.973763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.973735    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs"]
Oct 30 20:01:03.974245 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:01:03.974202    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded1b2eb5_91e7_4cf8_8bbb_f9eb043b0c1b.slice/crio-3a963a47348996b1f51451dd8735b2353b2e19bf7638f152e82c2b541d0d1629 WatchSource:0}: Error finding container 3a963a47348996b1f51451dd8735b2353b2e19bf7638f152e82c2b541d0d1629: Status 404 returned error can't find the container with id 3a963a47348996b1f51451dd8735b2353b2e19bf7638f152e82c2b541d0d1629
Oct 30 20:01:03.975275 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:01:03.975244    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c52e04b_b1aa_4f44_a01e_0cdef1cdef9f.slice/crio-9b492aa5cb9ef22495d004228fbce436b61c0ef2c1428dde676f9f8ce9761fdd WatchSource:0}: Error finding container 9b492aa5cb9ef22495d004228fbce436b61c0ef2c1428dde676f9f8ce9761fdd: Status 404 returned error can't find the container with id 9b492aa5cb9ef22495d004228fbce436b61c0ef2c1428dde676f9f8ce9761fdd
Oct 30 20:01:03.978109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:03.978087    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2"]
Oct 30 20:01:04.204877 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:04.204842    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs" event={"ID":"4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f","Type":"ContainerStarted","Data":"80f2d94c704d44451b94ae9540ad792820012081e7afa32521cd64681193d32a"}
Oct 30 20:01:04.205256 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:04.204912    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs" event={"ID":"4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f","Type":"ContainerStarted","Data":"9b492aa5cb9ef22495d004228fbce436b61c0ef2c1428dde676f9f8ce9761fdd"}
Oct 30 20:01:04.205256 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:04.205060    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs"
Oct 30 20:01:04.206699 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:04.206674    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2" event={"ID":"ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b","Type":"ContainerStarted","Data":"1df72a5c76e29538ace0fb80dcca9f44031a3f84e4e20e865465340fbe6e0a39"}
Oct 30 20:01:04.206800 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:04.206721    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2" event={"ID":"ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b","Type":"ContainerStarted","Data":"3a963a47348996b1f51451dd8735b2353b2e19bf7638f152e82c2b541d0d1629"}
Oct 30 20:01:04.206800 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:04.206751    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2"
Oct 30 20:01:04.231095 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:04.230977    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs" podStartSLOduration=1.230964341 podStartE2EDuration="1.230964341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:04.2296041 +0000 UTC m=+1818.813873718" watchObservedRunningTime="2026-10-30 20:01:04.230964341 +0000 UTC m=+1818.815233957"
Oct 30 20:01:04.255225 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:04.255183    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2" podStartSLOduration=1.255168648 podStartE2EDuration="1.255168648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:04.250951448 +0000 UTC m=+1818.835221065" watchObservedRunningTime="2026-10-30 20:01:04.255168648 +0000 UTC m=+1818.839438268"
Oct 30 20:01:06.215363 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:06.215335    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-ab1424"
Oct 30 20:01:06.218323 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:01:06.218290    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11080-ab1424\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3" pod="opendatahub/test-curl-11080-ab1424"
Oct 30 20:01:06.247629 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:06.247598    5408 generic.go:356] "Generic (PLEG): container finished" podID="aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3" containerID="50fc0c9e67f5964ce07f7c5954ba7f2699790ce9ddd144d99ab98466305fc56c" exitCode=137
Oct 30 20:01:06.247757 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:06.247639    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11080-ab1424"
Oct 30 20:01:06.247757 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:06.247674    5408 scope.go:122] "RemoveContainer" containerID="50fc0c9e67f5964ce07f7c5954ba7f2699790ce9ddd144d99ab98466305fc56c"
Oct 30 20:01:06.251107 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:01:06.251081    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11080-ab1424\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3" pod="opendatahub/test-curl-11080-ab1424"
Oct 30 20:01:06.257673 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:06.257650    5408 scope.go:122] "RemoveContainer" containerID="50fc0c9e67f5964ce07f7c5954ba7f2699790ce9ddd144d99ab98466305fc56c"
Oct 30 20:01:06.257956 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:01:06.257934    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50fc0c9e67f5964ce07f7c5954ba7f2699790ce9ddd144d99ab98466305fc56c\": container with ID starting with 50fc0c9e67f5964ce07f7c5954ba7f2699790ce9ddd144d99ab98466305fc56c not found: ID does not exist" containerID="50fc0c9e67f5964ce07f7c5954ba7f2699790ce9ddd144d99ab98466305fc56c"
Oct 30 20:01:06.258011 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:06.257965    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50fc0c9e67f5964ce07f7c5954ba7f2699790ce9ddd144d99ab98466305fc56c"} err="failed to get container status \"50fc0c9e67f5964ce07f7c5954ba7f2699790ce9ddd144d99ab98466305fc56c\": rpc error: code = NotFound desc = could not find container \"50fc0c9e67f5964ce07f7c5954ba7f2699790ce9ddd144d99ab98466305fc56c\": container with ID starting with 50fc0c9e67f5964ce07f7c5954ba7f2699790ce9ddd144d99ab98466305fc56c not found: ID does not exist"
Oct 30 20:01:06.370796 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:06.370718    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3-kube-api-access-klj6t\" (UniqueName: \"kubernetes.io/projected/aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3-kube-api-access-klj6t\") pod \"aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3\" (UID: \"aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3\") "
Oct 30 20:01:06.372810 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:06.372777    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3-kube-api-access-klj6t" pod "aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3" (UID: "aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3"). InnerVolumeSpecName "kube-api-access-klj6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:06.473551 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:06.473515    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klj6t\" (UniqueName: \"kubernetes.io/projected/aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3-kube-api-access-klj6t\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:06.559944 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:01:06.559889    5408 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11080-ab1424\" is forbidden: User \"system:node:ip-10-0-141-19.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-19.us-east-2.compute.internal' and this object" podUID="aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3" pod="opendatahub/test-curl-11080-ab1424"
Oct 30 20:01:07.534559 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:07.534526    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3" path="/var/lib/kubelet/pods/aa8d6bf1-0ce4-4a28-be7a-8ec300aee1b3/volumes"
Oct 30 20:01:09.661683 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:01:09.661650    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:01:15.242333 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:15.242299    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2"
Oct 30 20:01:15.243117 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:15.243098    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs"
Oct 30 20:01:16.159303 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:16.159272    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp"]
Oct 30 20:01:20.055054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.052560    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq"]
Oct 30 20:01:20.064062 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.059133    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq"
Oct 30 20:01:20.066362 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.066334    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-96ec93\""
Oct 30 20:01:20.066952 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.066931    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs"]
Oct 30 20:01:20.076233 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.076212    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq"]
Oct 30 20:01:20.076473 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.076457    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs"
Oct 30 20:01:20.077501 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.077480    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs"]
Oct 30 20:01:20.237025 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.236983    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6458172-491b-41c3-a5a1-d8acc7652807-plugins-config-volume\") pod \"payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq\" (UID: \"d6458172-491b-41c3-a5a1-d8acc7652807\") " pod="openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq"
Oct 30 20:01:20.237386 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.237361    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcdzq\" (UniqueName: \"kubernetes.io/projected/d8634c36-96d5-4ce4-b82f-fd4fd7a6b481-kube-api-access-mcdzq\") pod \"payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs\" (UID: \"d8634c36-96d5-4ce4-b82f-fd4fd7a6b481\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs"
Oct 30 20:01:20.237548 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.237533    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8634c36-96d5-4ce4-b82f-fd4fd7a6b481-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs\" (UID: \"d8634c36-96d5-4ce4-b82f-fd4fd7a6b481\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs"
Oct 30 20:01:20.239525 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.239428    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chxbv\" (UniqueName: \"kubernetes.io/projected/d6458172-491b-41c3-a5a1-d8acc7652807-kube-api-access-chxbv\") pod \"payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq\" (UID: \"d6458172-491b-41c3-a5a1-d8acc7652807\") " pod="openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq"
Oct 30 20:01:20.340992 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.340909    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chxbv\" (UniqueName: \"kubernetes.io/projected/d6458172-491b-41c3-a5a1-d8acc7652807-kube-api-access-chxbv\") pod \"payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq\" (UID: \"d6458172-491b-41c3-a5a1-d8acc7652807\") " pod="openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq"
Oct 30 20:01:20.341350 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.341332    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6458172-491b-41c3-a5a1-d8acc7652807-plugins-config-volume\") pod \"payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq\" (UID: \"d6458172-491b-41c3-a5a1-d8acc7652807\") " pod="openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq"
Oct 30 20:01:20.341414 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.341388    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcdzq\" (UniqueName: \"kubernetes.io/projected/d8634c36-96d5-4ce4-b82f-fd4fd7a6b481-kube-api-access-mcdzq\") pod \"payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs\" (UID: \"d8634c36-96d5-4ce4-b82f-fd4fd7a6b481\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs"
Oct 30 20:01:20.341454 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.341424    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8634c36-96d5-4ce4-b82f-fd4fd7a6b481-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs\" (UID: \"d8634c36-96d5-4ce4-b82f-fd4fd7a6b481\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs"
Oct 30 20:01:20.342297 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.342268    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8634c36-96d5-4ce4-b82f-fd4fd7a6b481-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs\" (UID: \"d8634c36-96d5-4ce4-b82f-fd4fd7a6b481\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs"
Oct 30 20:01:20.342416 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.342276    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6458172-491b-41c3-a5a1-d8acc7652807-plugins-config-volume\") pod \"payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq\" (UID: \"d6458172-491b-41c3-a5a1-d8acc7652807\") " pod="openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq"
Oct 30 20:01:20.408054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.403771    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcdzq\" (UniqueName: \"kubernetes.io/projected/d8634c36-96d5-4ce4-b82f-fd4fd7a6b481-kube-api-access-mcdzq\") pod \"payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs\" (UID: \"d8634c36-96d5-4ce4-b82f-fd4fd7a6b481\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs"
Oct 30 20:01:20.423052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.414810    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chxbv\" (UniqueName: \"kubernetes.io/projected/d6458172-491b-41c3-a5a1-d8acc7652807-kube-api-access-chxbv\") pod \"payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq\" (UID: \"d6458172-491b-41c3-a5a1-d8acc7652807\") " pod="openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq"
Oct 30 20:01:20.685903 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.685813    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq"
Oct 30 20:01:20.741107 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:20.741069    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs"
Oct 30 20:01:21.159927 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.159873    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp" podUID="a69dfb51-e759-48f5-9a34-f1047bc4c2c2" containerName="payload-processing" containerID="cri-o://899f9f4eff92120e4ecda0530fe8269e99de0751b704671095406973351495e5" gracePeriod=25
Oct 30 20:01:21.231656 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:01:21.231619    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8634c36_96d5_4ce4_b82f_fd4fd7a6b481.slice/crio-9c64dd1e272ad86a8609fd5fa777c933b7b6ba344a2a5a8c5ebfa109fb2bc709 WatchSource:0}: Error finding container 9c64dd1e272ad86a8609fd5fa777c933b7b6ba344a2a5a8c5ebfa109fb2bc709: Status 404 returned error can't find the container with id 9c64dd1e272ad86a8609fd5fa777c933b7b6ba344a2a5a8c5ebfa109fb2bc709
Oct 30 20:01:21.232459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.232435    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs"]
Oct 30 20:01:21.232577 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:01:21.232451    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6458172_491b_41c3_a5a1_d8acc7652807.slice/crio-a3e5755cd94c1ba62dc438541bdc24214f4ad97eb8a035406caf8b009a7c3dd1 WatchSource:0}: Error finding container a3e5755cd94c1ba62dc438541bdc24214f4ad97eb8a035406caf8b009a7c3dd1: Status 404 returned error can't find the container with id a3e5755cd94c1ba62dc438541bdc24214f4ad97eb8a035406caf8b009a7c3dd1
Oct 30 20:01:21.235911 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.235890    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq"]
Oct 30 20:01:21.477039 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.477002    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp"
Oct 30 20:01:21.480134 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.480065    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs" event={"ID":"d8634c36-96d5-4ce4-b82f-fd4fd7a6b481","Type":"ContainerStarted","Data":"72fcc56292d3f54cc511729f8ab940f35ffc19f7deee410e1e23aa3cbfc4d001"}
Oct 30 20:01:21.480254 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.480132    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs"
Oct 30 20:01:21.480254 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.480155    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs" event={"ID":"d8634c36-96d5-4ce4-b82f-fd4fd7a6b481","Type":"ContainerStarted","Data":"9c64dd1e272ad86a8609fd5fa777c933b7b6ba344a2a5a8c5ebfa109fb2bc709"}
Oct 30 20:01:21.481720 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.481698    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq" event={"ID":"d6458172-491b-41c3-a5a1-d8acc7652807","Type":"ContainerStarted","Data":"fb4ba76c00f95fb985c381dd309f53a30654e6ba8b7ab9b2d007abcf33da5b8f"}
Oct 30 20:01:21.481843 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.481737    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq" event={"ID":"d6458172-491b-41c3-a5a1-d8acc7652807","Type":"ContainerStarted","Data":"a3e5755cd94c1ba62dc438541bdc24214f4ad97eb8a035406caf8b009a7c3dd1"}
Oct 30 20:01:21.481843 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.481765    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq"
Oct 30 20:01:21.483045 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.483013    5408 generic.go:356] "Generic (PLEG): container finished" podID="a69dfb51-e759-48f5-9a34-f1047bc4c2c2" containerID="899f9f4eff92120e4ecda0530fe8269e99de0751b704671095406973351495e5" exitCode=0
Oct 30 20:01:21.483154 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.483068    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp" event={"ID":"a69dfb51-e759-48f5-9a34-f1047bc4c2c2","Type":"ContainerDied","Data":"899f9f4eff92120e4ecda0530fe8269e99de0751b704671095406973351495e5"}
Oct 30 20:01:21.483154 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.483061    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp"
Oct 30 20:01:21.483154 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.483114    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp" event={"ID":"a69dfb51-e759-48f5-9a34-f1047bc4c2c2","Type":"ContainerDied","Data":"cb96bbb323c9b518119986d266e123f2186b2c283b1b5edd92f6f8e4f30e89fa"}
Oct 30 20:01:21.483154 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.483145    5408 scope.go:122] "RemoveContainer" containerID="899f9f4eff92120e4ecda0530fe8269e99de0751b704671095406973351495e5"
Oct 30 20:01:21.494235 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.494216    5408 scope.go:122] "RemoveContainer" containerID="899f9f4eff92120e4ecda0530fe8269e99de0751b704671095406973351495e5"
Oct 30 20:01:21.494483 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:01:21.494466    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"899f9f4eff92120e4ecda0530fe8269e99de0751b704671095406973351495e5\": container with ID starting with 899f9f4eff92120e4ecda0530fe8269e99de0751b704671095406973351495e5 not found: ID does not exist" containerID="899f9f4eff92120e4ecda0530fe8269e99de0751b704671095406973351495e5"
Oct 30 20:01:21.494532 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.494490    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"899f9f4eff92120e4ecda0530fe8269e99de0751b704671095406973351495e5"} err="failed to get container status \"899f9f4eff92120e4ecda0530fe8269e99de0751b704671095406973351495e5\": rpc error: code = NotFound desc = could not find container \"899f9f4eff92120e4ecda0530fe8269e99de0751b704671095406973351495e5\": container with ID starting with 899f9f4eff92120e4ecda0530fe8269e99de0751b704671095406973351495e5 not found: ID does not exist"
Oct 30 20:01:21.545666 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.537262    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a69dfb51-e759-48f5-9a34-f1047bc4c2c2-kube-api-access-ph8xg\" (UniqueName: \"kubernetes.io/projected/a69dfb51-e759-48f5-9a34-f1047bc4c2c2-kube-api-access-ph8xg\") pod \"a69dfb51-e759-48f5-9a34-f1047bc4c2c2\" (UID: \"a69dfb51-e759-48f5-9a34-f1047bc4c2c2\") "
Oct 30 20:01:21.545666 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.537528    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a69dfb51-e759-48f5-9a34-f1047bc4c2c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a69dfb51-e759-48f5-9a34-f1047bc4c2c2-plugins-config-volume\") pod \"a69dfb51-e759-48f5-9a34-f1047bc4c2c2\" (UID: \"a69dfb51-e759-48f5-9a34-f1047bc4c2c2\") "
Oct 30 20:01:21.548159 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.548125    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a69dfb51-e759-48f5-9a34-f1047bc4c2c2-plugins-config-volume" pod "a69dfb51-e759-48f5-9a34-f1047bc4c2c2" (UID: "a69dfb51-e759-48f5-9a34-f1047bc4c2c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:21.558114 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.558018    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69dfb51-e759-48f5-9a34-f1047bc4c2c2-kube-api-access-ph8xg" pod "a69dfb51-e759-48f5-9a34-f1047bc4c2c2" (UID: "a69dfb51-e759-48f5-9a34-f1047bc4c2c2"). InnerVolumeSpecName "kube-api-access-ph8xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:21.589348 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.589296    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq" podStartSLOduration=2.589279125 podStartE2EDuration="2.589279125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:21.584900917 +0000 UTC m=+1836.169170533" watchObservedRunningTime="2026-10-30 20:01:21.589279125 +0000 UTC m=+1836.173548743"
Oct 30 20:01:21.622439 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.622394    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs" podStartSLOduration=2.622380283 podStartE2EDuration="2.622380283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:21.620258858 +0000 UTC m=+1836.204528472" watchObservedRunningTime="2026-10-30 20:01:21.622380283 +0000 UTC m=+1836.206649903"
Oct 30 20:01:21.640952 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.640919    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph8xg\" (UniqueName: \"kubernetes.io/projected/a69dfb51-e759-48f5-9a34-f1047bc4c2c2-kube-api-access-ph8xg\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:21.640952 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.640949    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a69dfb51-e759-48f5-9a34-f1047bc4c2c2-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:21.856239 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.856202    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp"]
Oct 30 20:01:21.860956 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:21.860907    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c4c072-666d544fc4-gfgqp"]
Oct 30 20:01:23.628054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:23.616305    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a69dfb51-e759-48f5-9a34-f1047bc4c2c2" path="/var/lib/kubelet/pods/a69dfb51-e759-48f5-9a34-f1047bc4c2c2/volumes"
Oct 30 20:01:26.193330 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:26.193287    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"]
Oct 30 20:01:32.494330 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:32.494299    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq"
Oct 30 20:01:32.494706 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:32.494379    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs"
Oct 30 20:01:38.133179 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.133127    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"]
Oct 30 20:01:38.137314 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.137283    5408 cpu_manager.go:414] "containerMap: removing container" podUID="a69dfb51-e759-48f5-9a34-f1047bc4c2c2" containerName="payload-processing"
Oct 30 20:01:38.137568 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.137537    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a69dfb51-e759-48f5-9a34-f1047bc4c2c2" containerName="payload-processing"
Oct 30 20:01:38.137920 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.137893    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="a69dfb51-e759-48f5-9a34-f1047bc4c2c2" containerName="payload-processing"
Oct 30 20:01:38.146772 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.146747    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.160585 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.160432    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"]
Oct 30 20:01:38.168098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.168069    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-podinfo\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.168200 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.168136    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istiod-ca-cert\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.168200 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.168179    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-envoy\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.168287 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.168220    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-token\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.168371 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.168359    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-workload-certs\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.168410 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.168382    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2242\" (UniqueName: \"kubernetes.io/projected/96a3152c-afa9-4f5f-9fb5-ea84606ff044-kube-api-access-r2242\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.168443 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.168425    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-data\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.168474 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.168463    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-workload-socket\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.168508 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.168477    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-credential-socket\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.213480 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.213448    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c50e1e-openshift-default-dockercfg-wkgrj\""
Oct 30 20:01:38.269567 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.269533    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-workload-certs\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.269567 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.269568    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2242\" (UniqueName: \"kubernetes.io/projected/96a3152c-afa9-4f5f-9fb5-ea84606ff044-kube-api-access-r2242\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.269782 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.269696    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-data\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.269782 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.269748    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-workload-socket\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.269782 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.269775    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-credential-socket\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.269975 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.269937    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-podinfo\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.270054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.270025    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istiod-ca-cert\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.270118 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.269946    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-workload-certs\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.270118 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.270104    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-envoy\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.270118 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.270110    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-credential-socket\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.270288 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.270091    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-data\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.270288 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.270173    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-workload-socket\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.270288 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.270194    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-token\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.270737 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.270714    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istiod-ca-cert\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.272308 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.272285    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-envoy\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.272647 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.272628    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-podinfo\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.287149 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.287129    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-token\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.290501 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.290479    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2242\" (UniqueName: \"kubernetes.io/projected/96a3152c-afa9-4f5f-9fb5-ea84606ff044-kube-api-access-r2242\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.473884 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.473807    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:38.773269 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:01:38.773234    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a3152c_afa9_4f5f_9fb5_ea84606ff044.slice/crio-a9b0fa37f889787ae2c177730eb67ac550ea0aaa591ad941ac0785551ab55387 WatchSource:0}: Error finding container a9b0fa37f889787ae2c177730eb67ac550ea0aaa591ad941ac0785551ab55387: Status 404 returned error can't find the container with id a9b0fa37f889787ae2c177730eb67ac550ea0aaa591ad941ac0785551ab55387
Oct 30 20:01:38.773398 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.773286    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"]
Oct 30 20:01:38.777063 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.777004    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:01:38.777137 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.777111    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:01:38.777181 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.777159    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:01:38.833437 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:38.833409    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4" event={"ID":"96a3152c-afa9-4f5f-9fb5-ea84606ff044","Type":"ContainerStarted","Data":"a9b0fa37f889787ae2c177730eb67ac550ea0aaa591ad941ac0785551ab55387"}
Oct 30 20:01:39.663197 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:01:39.663157    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:01:39.842719 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:39.842679    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4" event={"ID":"96a3152c-afa9-4f5f-9fb5-ea84606ff044","Type":"ContainerStarted","Data":"d96e51c5327939728b9ef234c6617463dc4da925980a393fa308d160f9a958eb"}
Oct 30 20:01:39.920086 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:39.919059    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4" podStartSLOduration=1.918962623 podStartE2EDuration="1.918962623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:39.87769951 +0000 UTC m=+1854.461969151" watchObservedRunningTime="2026-10-30 20:01:39.918962623 +0000 UTC m=+1854.503232268"
Oct 30 20:01:40.474503 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:40.474469    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:40.475805 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:40.475781    5408 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 20:01:40.475916 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:40.475830    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4" podUID="96a3152c-afa9-4f5f-9fb5-ea84606ff044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 20:01:41.255702 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:41.255653    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p" podUID="f96b2a7d-14e7-4386-921d-b3bb1d6ec55d" containerName="main" containerID="cri-o://356f79db6ea832ee9dff29d7c43557d7fe380404971f61d9984be480be2ac48e" gracePeriod=45
Oct 30 20:01:41.490936 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:41.480169    5408 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 20:01:41.490936 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:41.480236    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4" podUID="96a3152c-afa9-4f5f-9fb5-ea84606ff044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 20:01:42.095348 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.094294    5408 generic.go:356] "Generic (PLEG): container finished" podID="f96b2a7d-14e7-4386-921d-b3bb1d6ec55d" containerID="356f79db6ea832ee9dff29d7c43557d7fe380404971f61d9984be480be2ac48e" exitCode=0
Oct 30 20:01:42.095348 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.094743    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p" event={"ID":"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d","Type":"ContainerDied","Data":"356f79db6ea832ee9dff29d7c43557d7fe380404971f61d9984be480be2ac48e"}
Oct 30 20:01:42.143521 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.143490    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 20:01:42.193787 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.193759    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-kube-api-access-rlw7g\" (UniqueName: \"kubernetes.io/projected/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-kube-api-access-rlw7g\") pod \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") "
Oct 30 20:01:42.193953 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.193854    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-tmp-dir\") pod \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") "
Oct 30 20:01:42.194018 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.193955    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-dshm\") pod \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") "
Oct 30 20:01:42.194094 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.194015    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-model-cache\") pod \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") "
Oct 30 20:01:42.194094 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.194066    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-tls-certs\" (UniqueName: \"kubernetes.io/secret/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-tls-certs\") pod \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") "
Oct 30 20:01:42.194223 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.194195    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-tmp-dir" pod "f96b2a7d-14e7-4386-921d-b3bb1d6ec55d" (UID: "f96b2a7d-14e7-4386-921d-b3bb1d6ec55d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:42.194431 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.194409    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-home\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-home\") pod \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\" (UID: \"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d\") "
Oct 30 20:01:42.195728 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.194665    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-model-cache" pod "f96b2a7d-14e7-4386-921d-b3bb1d6ec55d" (UID: "f96b2a7d-14e7-4386-921d-b3bb1d6ec55d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:42.195728 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.194854    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-home" pod "f96b2a7d-14e7-4386-921d-b3bb1d6ec55d" (UID: "f96b2a7d-14e7-4386-921d-b3bb1d6ec55d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:42.196311 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.196270    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:42.196765 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.196747    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:42.196843 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.196768    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:42.197436 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.197384    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-kube-api-access-rlw7g" pod "f96b2a7d-14e7-4386-921d-b3bb1d6ec55d" (UID: "f96b2a7d-14e7-4386-921d-b3bb1d6ec55d"). InnerVolumeSpecName "kube-api-access-rlw7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:42.197536 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.197472    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-dshm" pod "f96b2a7d-14e7-4386-921d-b3bb1d6ec55d" (UID: "f96b2a7d-14e7-4386-921d-b3bb1d6ec55d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:42.198798 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.198777    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-tls-certs" pod "f96b2a7d-14e7-4386-921d-b3bb1d6ec55d" (UID: "f96b2a7d-14e7-4386-921d-b3bb1d6ec55d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:42.298617 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.298582    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlw7g\" (UniqueName: \"kubernetes.io/projected/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-kube-api-access-rlw7g\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:42.298617 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.298614    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:42.298617 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.298623    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:42.378316 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.378224    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"]
Oct 30 20:01:42.378744 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.378706    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml" podUID="2d3a55c8-98fe-4911-b3c5-476a14bc809a" containerName="istio-proxy" containerID="cri-o://51a2f0fab52732b11f2ce8b7ebae38bedff6c786d990b19419cff21d5f8f9dba" gracePeriod=30
Oct 30 20:01:42.501840 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.501802    5408 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 20:01:42.502109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:42.502077    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4" podUID="96a3152c-afa9-4f5f-9fb5-ea84606ff044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 20:01:43.016091 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:43.012425    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p" podUID="f96b2a7d-14e7-4386-921d-b3bb1d6ec55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:43.108930 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:43.108894    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p" event={"ID":"f96b2a7d-14e7-4386-921d-b3bb1d6ec55d","Type":"ContainerDied","Data":"d4df1f2804394b4b2afbb4f03311aa1d27ea22e75b508646c3fbbd6c69624c02"}
Oct 30 20:01:43.109448 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:43.109429    5408 scope.go:122] "RemoveContainer" containerID="356f79db6ea832ee9dff29d7c43557d7fe380404971f61d9984be480be2ac48e"
Oct 30 20:01:43.109727 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:43.109237    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"
Oct 30 20:01:43.163414 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:43.163334    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"]
Oct 30 20:01:43.167684 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:43.167661    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ae43d70e/e2e-lifecycle-model-ae43d70e-kserve-648fbc48f7-ghf7p"]
Oct 30 20:01:43.476754 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:43.476667    5408 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 20:01:43.477356 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:43.477319    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4" podUID="96a3152c-afa9-4f5f-9fb5-ea84606ff044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 20:01:43.602055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:43.589159    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f96b2a7d-14e7-4386-921d-b3bb1d6ec55d" path="/var/lib/kubelet/pods/f96b2a7d-14e7-4386-921d-b3bb1d6ec55d/volumes"
Oct 30 20:01:44.543723 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:44.543682    5408 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 20:01:44.544258 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:44.543755    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4" podUID="96a3152c-afa9-4f5f-9fb5-ea84606ff044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 20:01:45.525054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:45.520297    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:45.657054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:45.654802    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:45.657054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:45.654896    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:01:47.754963 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.754934    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 20:01:47.867256 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.867163    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-credential-socket\") pod \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") "
Oct 30 20:01:47.867256 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.867250    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-token\" (UniqueName: \"kubernetes.io/projected/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-token\") pod \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") "
Oct 30 20:01:47.867485 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.867323    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-podinfo\") pod \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") "
Oct 30 20:01:47.867485 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.867358    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-workload-socket\") pod \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") "
Oct 30 20:01:47.867485 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.867396    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-envoy\") pod \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") "
Oct 30 20:01:47.867643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.867482    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-credential-socket" pod "2d3a55c8-98fe-4911-b3c5-476a14bc809a" (UID: "2d3a55c8-98fe-4911-b3c5-476a14bc809a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:47.867643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.867492    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-data\") pod \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") "
Oct 30 20:01:47.867643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.867599    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istiod-ca-cert\") pod \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") "
Oct 30 20:01:47.867803 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.867746    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-data" pod "2d3a55c8-98fe-4911-b3c5-476a14bc809a" (UID: "2d3a55c8-98fe-4911-b3c5-476a14bc809a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:47.867865 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.867804    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-workload-socket" pod "2d3a55c8-98fe-4911-b3c5-476a14bc809a" (UID: "2d3a55c8-98fe-4911-b3c5-476a14bc809a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:47.867865 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.867814    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-workload-certs\") pod \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") "
Oct 30 20:01:47.868188 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.868161    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istiod-ca-cert" pod "2d3a55c8-98fe-4911-b3c5-476a14bc809a" (UID: "2d3a55c8-98fe-4911-b3c5-476a14bc809a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:47.868314 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.868136    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d3a55c8-98fe-4911-b3c5-476a14bc809a-kube-api-access-gw9rs\" (UniqueName: \"kubernetes.io/projected/2d3a55c8-98fe-4911-b3c5-476a14bc809a-kube-api-access-gw9rs\") pod \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\" (UID: \"2d3a55c8-98fe-4911-b3c5-476a14bc809a\") "
Oct 30 20:01:47.868381 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.868360    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-workload-certs" pod "2d3a55c8-98fe-4911-b3c5-476a14bc809a" (UID: "2d3a55c8-98fe-4911-b3c5-476a14bc809a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:47.868971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.868948    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:47.868971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.868970    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:47.869180 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.868980    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:47.869180 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.868989    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:47.869180 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.868998    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:47.870179 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.870151    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-podinfo" pod "2d3a55c8-98fe-4911-b3c5-476a14bc809a" (UID: "2d3a55c8-98fe-4911-b3c5-476a14bc809a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:47.870283 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.870155    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-token" pod "2d3a55c8-98fe-4911-b3c5-476a14bc809a" (UID: "2d3a55c8-98fe-4911-b3c5-476a14bc809a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:47.870420 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.870405    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-envoy" pod "2d3a55c8-98fe-4911-b3c5-476a14bc809a" (UID: "2d3a55c8-98fe-4911-b3c5-476a14bc809a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:47.870484 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.870453    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3a55c8-98fe-4911-b3c5-476a14bc809a-kube-api-access-gw9rs" pod "2d3a55c8-98fe-4911-b3c5-476a14bc809a" (UID: "2d3a55c8-98fe-4911-b3c5-476a14bc809a"). InnerVolumeSpecName "kube-api-access-gw9rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:47.970484 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.970451    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:47.970484 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.970482    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:47.970484 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.970491    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw9rs\" (UniqueName: \"kubernetes.io/projected/2d3a55c8-98fe-4911-b3c5-476a14bc809a-kube-api-access-gw9rs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:47.970722 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:47.970502    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d3a55c8-98fe-4911-b3c5-476a14bc809a-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:48.258758 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:48.258725    5408 generic.go:356] "Generic (PLEG): container finished" podID="2d3a55c8-98fe-4911-b3c5-476a14bc809a" containerID="51a2f0fab52732b11f2ce8b7ebae38bedff6c786d990b19419cff21d5f8f9dba" exitCode=0
Oct 30 20:01:48.258945 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:48.258773    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"
Oct 30 20:01:48.258945 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:48.258808    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml" event={"ID":"2d3a55c8-98fe-4911-b3c5-476a14bc809a","Type":"ContainerDied","Data":"51a2f0fab52732b11f2ce8b7ebae38bedff6c786d990b19419cff21d5f8f9dba"}
Oct 30 20:01:48.258945 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:48.258863    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml" event={"ID":"2d3a55c8-98fe-4911-b3c5-476a14bc809a","Type":"ContainerDied","Data":"8d4d00a91146b60c238ae868b235e643a252a4d1db0157af7e91e5d117415c11"}
Oct 30 20:01:48.258945 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:48.258887    5408 scope.go:122] "RemoveContainer" containerID="51a2f0fab52732b11f2ce8b7ebae38bedff6c786d990b19419cff21d5f8f9dba"
Oct 30 20:01:48.271144 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:48.271095    5408 scope.go:122] "RemoveContainer" containerID="51a2f0fab52732b11f2ce8b7ebae38bedff6c786d990b19419cff21d5f8f9dba"
Oct 30 20:01:48.271439 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:01:48.271418    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a2f0fab52732b11f2ce8b7ebae38bedff6c786d990b19419cff21d5f8f9dba\": container with ID starting with 51a2f0fab52732b11f2ce8b7ebae38bedff6c786d990b19419cff21d5f8f9dba not found: ID does not exist" containerID="51a2f0fab52732b11f2ce8b7ebae38bedff6c786d990b19419cff21d5f8f9dba"
Oct 30 20:01:48.271503 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:48.271446    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a2f0fab52732b11f2ce8b7ebae38bedff6c786d990b19419cff21d5f8f9dba"} err="failed to get container status \"51a2f0fab52732b11f2ce8b7ebae38bedff6c786d990b19419cff21d5f8f9dba\": rpc error: code = NotFound desc = could not find container \"51a2f0fab52732b11f2ce8b7ebae38bedff6c786d990b19419cff21d5f8f9dba\": container with ID starting with 51a2f0fab52732b11f2ce8b7ebae38bedff6c786d990b19419cff21d5f8f9dba not found: ID does not exist"
Oct 30 20:01:48.291281 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:48.291255    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"]
Oct 30 20:01:48.297051 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:48.297016    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vnhml"]
Oct 30 20:01:49.536063 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:49.536004    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d3a55c8-98fe-4911-b3c5-476a14bc809a" path="/var/lib/kubelet/pods/2d3a55c8-98fe-4911-b3c5-476a14bc809a/volumes"
Oct 30 20:01:58.003107 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:58.003052    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"]
Oct 30 20:01:58.003614 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:58.003429    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf" podUID="8b8025ef-9bd4-463a-bc56-50d8d46673c2" containerName="istio-proxy" containerID="cri-o://f2ff12c85c16935d05433a328443cb28a184fb0085b4a5f1bb079f59bc6dc24c" gracePeriod=30
Oct 30 20:01:58.009422 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:58.009395    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"]
Oct 30 20:01:58.009680 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:01:58.009622    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn" podUID="9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" containerName="istio-proxy" containerID="cri-o://e80db79493b688707ecb8b39bb6f64951de6049ad16542d3f4de9c90b0bde2d2" gracePeriod=30
Oct 30 20:02:03.482398 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.482373    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 20:02:03.499826 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.499807    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 20:02:03.628144 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.628109    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-podinfo\") pod \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") "
Oct 30 20:02:03.628357 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.628334    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-data\") pod \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") "
Oct 30 20:02:03.628438 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.628401    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-workload-certs\") pod \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") "
Oct 30 20:02:03.628508 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.628438    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-kube-api-access-mgbc9\" (UniqueName: \"kubernetes.io/projected/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-kube-api-access-mgbc9\") pod \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") "
Oct 30 20:02:03.628508 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.628482    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-workload-certs\") pod \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") "
Oct 30 20:02:03.628607 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.628553    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-workload-socket\") pod \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") "
Oct 30 20:02:03.628607 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.628589    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istiod-ca-cert\") pod \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") "
Oct 30 20:02:03.628701 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.628637    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-data" pod "9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" (UID: "9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:03.628701 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.628639    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-token\" (UniqueName: \"kubernetes.io/projected/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-token\") pod \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") "
Oct 30 20:02:03.628801 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.628730    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-workload-certs" pod "8b8025ef-9bd4-463a-bc56-50d8d46673c2" (UID: "8b8025ef-9bd4-463a-bc56-50d8d46673c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:03.628801 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.628746    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-data\") pod \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") "
Oct 30 20:02:03.628975 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.628953    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-workload-certs" pod "9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" (UID: "9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:03.629076 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.629055    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-workload-socket" pod "8b8025ef-9bd4-463a-bc56-50d8d46673c2" (UID: "8b8025ef-9bd4-463a-bc56-50d8d46673c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:03.629330 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.629305    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-credential-socket\") pod \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") "
Oct 30 20:02:03.629423 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.629384    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-envoy\") pod \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") "
Oct 30 20:02:03.629480 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.629444    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istiod-ca-cert\") pod \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") "
Oct 30 20:02:03.629614 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.629591    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istiod-ca-cert" pod "8b8025ef-9bd4-463a-bc56-50d8d46673c2" (UID: "8b8025ef-9bd4-463a-bc56-50d8d46673c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:03.629744 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.629722    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-podinfo\") pod \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") "
Oct 30 20:02:03.629810 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.629764    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-credential-socket\") pod \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") "
Oct 30 20:02:03.629858 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.629823    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-envoy\") pod \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") "
Oct 30 20:02:03.629906 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.629872    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-workload-socket\") pod \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") "
Oct 30 20:02:03.629952 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.629925    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-token\" (UniqueName: \"kubernetes.io/projected/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-token\") pod \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\" (UID: \"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec\") "
Oct 30 20:02:03.630003 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.629967    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-credential-socket" pod "9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" (UID: "9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:03.630003 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.629987    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8025ef-9bd4-463a-bc56-50d8d46673c2-kube-api-access-j52d2\" (UniqueName: \"kubernetes.io/projected/8b8025ef-9bd4-463a-bc56-50d8d46673c2-kube-api-access-j52d2\") pod \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\" (UID: \"8b8025ef-9bd4-463a-bc56-50d8d46673c2\") "
Oct 30 20:02:03.630244 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.630223    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-data" pod "8b8025ef-9bd4-463a-bc56-50d8d46673c2" (UID: "8b8025ef-9bd4-463a-bc56-50d8d46673c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:03.630317 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.630260    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-credential-socket" pod "8b8025ef-9bd4-463a-bc56-50d8d46673c2" (UID: "8b8025ef-9bd4-463a-bc56-50d8d46673c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:03.631119 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.631096    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.631794 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.631413    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.631794 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.631437    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.631794 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.631451    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.631794 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.631465    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.631794 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.631482    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.631794 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.631495    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.631794 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.631509    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.631794 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.631578    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istiod-ca-cert" pod "9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" (UID: "9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:03.632274 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.631834    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-workload-socket" pod "9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" (UID: "9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:03.633543 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.633509    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-envoy" pod "8b8025ef-9bd4-463a-bc56-50d8d46673c2" (UID: "8b8025ef-9bd4-463a-bc56-50d8d46673c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:03.634184 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.634153    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-podinfo" pod "9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" (UID: "9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:03.634184 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.634166    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8025ef-9bd4-463a-bc56-50d8d46673c2-kube-api-access-j52d2" pod "8b8025ef-9bd4-463a-bc56-50d8d46673c2" (UID: "8b8025ef-9bd4-463a-bc56-50d8d46673c2"). InnerVolumeSpecName "kube-api-access-j52d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:03.634613 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.634580    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-envoy" pod "9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" (UID: "9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:03.635177 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.635160    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-token" pod "9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" (UID: "9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:03.635616 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.635600    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-podinfo" pod "8b8025ef-9bd4-463a-bc56-50d8d46673c2" (UID: "8b8025ef-9bd4-463a-bc56-50d8d46673c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:03.643213 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.638916    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-token" pod "8b8025ef-9bd4-463a-bc56-50d8d46673c2" (UID: "8b8025ef-9bd4-463a-bc56-50d8d46673c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:03.643213 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.640461    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-kube-api-access-mgbc9" pod "9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" (UID: "9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec"). InnerVolumeSpecName "kube-api-access-mgbc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:03.733735 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.733701    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.733735 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.733736    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.733916 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.733751    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.733916 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.733766    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j52d2\" (UniqueName: \"kubernetes.io/projected/8b8025ef-9bd4-463a-bc56-50d8d46673c2-kube-api-access-j52d2\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.733916 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.733781    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.733916 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.733795    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgbc9\" (UniqueName: \"kubernetes.io/projected/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-kube-api-access-mgbc9\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.733916 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.733808    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.733916 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.733822    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.733916 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.733835    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.733916 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.733848    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b8025ef-9bd4-463a-bc56-50d8d46673c2-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:03.850818 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.850782    5408 generic.go:356] "Generic (PLEG): container finished" podID="8b8025ef-9bd4-463a-bc56-50d8d46673c2" containerID="f2ff12c85c16935d05433a328443cb28a184fb0085b4a5f1bb079f59bc6dc24c" exitCode=0
Oct 30 20:02:03.850982 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.850852    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"
Oct 30 20:02:03.850982 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.850860    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf" event={"ID":"8b8025ef-9bd4-463a-bc56-50d8d46673c2","Type":"ContainerDied","Data":"f2ff12c85c16935d05433a328443cb28a184fb0085b4a5f1bb079f59bc6dc24c"}
Oct 30 20:02:03.850982 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.850939    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf" event={"ID":"8b8025ef-9bd4-463a-bc56-50d8d46673c2","Type":"ContainerDied","Data":"70cd9538a3631d7e6bf628b2b4bfd954e9fa0234e7dac7aa7a53f49dc2a50e33"}
Oct 30 20:02:03.850982 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.850964    5408 scope.go:122] "RemoveContainer" containerID="f2ff12c85c16935d05433a328443cb28a184fb0085b4a5f1bb079f59bc6dc24c"
Oct 30 20:02:03.852566 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.852540    5408 generic.go:356] "Generic (PLEG): container finished" podID="9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" containerID="e80db79493b688707ecb8b39bb6f64951de6049ad16542d3f4de9c90b0bde2d2" exitCode=0
Oct 30 20:02:03.852684 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.852610    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"
Oct 30 20:02:03.852684 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.852619    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn" event={"ID":"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec","Type":"ContainerDied","Data":"e80db79493b688707ecb8b39bb6f64951de6049ad16542d3f4de9c90b0bde2d2"}
Oct 30 20:02:03.852684 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.852665    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn" event={"ID":"9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec","Type":"ContainerDied","Data":"083ee319d2ebf93d5559c20db402a670f9b70cd27222ab654c72d5a364aa156a"}
Oct 30 20:02:03.863528 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.863188    5408 scope.go:122] "RemoveContainer" containerID="f2ff12c85c16935d05433a328443cb28a184fb0085b4a5f1bb079f59bc6dc24c"
Oct 30 20:02:03.863764 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:02:03.863743    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ff12c85c16935d05433a328443cb28a184fb0085b4a5f1bb079f59bc6dc24c\": container with ID starting with f2ff12c85c16935d05433a328443cb28a184fb0085b4a5f1bb079f59bc6dc24c not found: ID does not exist" containerID="f2ff12c85c16935d05433a328443cb28a184fb0085b4a5f1bb079f59bc6dc24c"
Oct 30 20:02:03.863867 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.863770    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ff12c85c16935d05433a328443cb28a184fb0085b4a5f1bb079f59bc6dc24c"} err="failed to get container status \"f2ff12c85c16935d05433a328443cb28a184fb0085b4a5f1bb079f59bc6dc24c\": rpc error: code = NotFound desc = could not find container \"f2ff12c85c16935d05433a328443cb28a184fb0085b4a5f1bb079f59bc6dc24c\": container with ID starting with f2ff12c85c16935d05433a328443cb28a184fb0085b4a5f1bb079f59bc6dc24c not found: ID does not exist"
Oct 30 20:02:03.863867 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.863790    5408 scope.go:122] "RemoveContainer" containerID="e80db79493b688707ecb8b39bb6f64951de6049ad16542d3f4de9c90b0bde2d2"
Oct 30 20:02:03.873650 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.873628    5408 scope.go:122] "RemoveContainer" containerID="e80db79493b688707ecb8b39bb6f64951de6049ad16542d3f4de9c90b0bde2d2"
Oct 30 20:02:03.873954 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:02:03.873925    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80db79493b688707ecb8b39bb6f64951de6049ad16542d3f4de9c90b0bde2d2\": container with ID starting with e80db79493b688707ecb8b39bb6f64951de6049ad16542d3f4de9c90b0bde2d2 not found: ID does not exist" containerID="e80db79493b688707ecb8b39bb6f64951de6049ad16542d3f4de9c90b0bde2d2"
Oct 30 20:02:03.874083 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.873962    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80db79493b688707ecb8b39bb6f64951de6049ad16542d3f4de9c90b0bde2d2"} err="failed to get container status \"e80db79493b688707ecb8b39bb6f64951de6049ad16542d3f4de9c90b0bde2d2\": rpc error: code = NotFound desc = could not find container \"e80db79493b688707ecb8b39bb6f64951de6049ad16542d3f4de9c90b0bde2d2\": container with ID starting with e80db79493b688707ecb8b39bb6f64951de6049ad16542d3f4de9c90b0bde2d2 not found: ID does not exist"
Oct 30 20:02:03.896501 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.896447    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"]
Oct 30 20:02:03.925649 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.925617    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-jxppf"]
Oct 30 20:02:03.931690 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.931656    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"]
Oct 30 20:02:03.931961 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.931932    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5" podUID="54ccdd75-9a27-4c5a-9a0a-9277ec782195" containerName="maas-api" containerID="cri-o://ea64e23030a1ddc74df8537e77dd4c558b37f11288132e0e01af17a2bc2413d3" gracePeriod=30
Oct 30 20:02:03.953954 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.953919    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"]
Oct 30 20:02:03.973352 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:03.973318    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-z22qn"]
Oct 30 20:02:04.656987 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.656956    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 20:02:04.756130 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.756097    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ccdd75-9a27-4c5a-9a0a-9277ec782195-kube-api-access-5vgvj\" (UniqueName: \"kubernetes.io/projected/54ccdd75-9a27-4c5a-9a0a-9277ec782195-kube-api-access-5vgvj\") pod \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\" (UID: \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\") "
Oct 30 20:02:04.756303 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.756156    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ccdd75-9a27-4c5a-9a0a-9277ec782195-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ccdd75-9a27-4c5a-9a0a-9277ec782195-openshift-service-ca\") pod \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\" (UID: \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\") "
Oct 30 20:02:04.756303 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.756255    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-maas-api-tls\") pod \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\" (UID: \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\") "
Oct 30 20:02:04.756433 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.756407    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-metrics-certs\") pod \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\" (UID: \"54ccdd75-9a27-4c5a-9a0a-9277ec782195\") "
Oct 30 20:02:04.756636 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.756578    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ccdd75-9a27-4c5a-9a0a-9277ec782195-openshift-service-ca" pod "54ccdd75-9a27-4c5a-9a0a-9277ec782195" (UID: "54ccdd75-9a27-4c5a-9a0a-9277ec782195"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:04.757166 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.757144    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ccdd75-9a27-4c5a-9a0a-9277ec782195-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:04.758411 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.758388    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-maas-api-tls" pod "54ccdd75-9a27-4c5a-9a0a-9277ec782195" (UID: "54ccdd75-9a27-4c5a-9a0a-9277ec782195"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:04.758499 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.758439    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ccdd75-9a27-4c5a-9a0a-9277ec782195-kube-api-access-5vgvj" pod "54ccdd75-9a27-4c5a-9a0a-9277ec782195" (UID: "54ccdd75-9a27-4c5a-9a0a-9277ec782195"). InnerVolumeSpecName "kube-api-access-5vgvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:04.758499 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.758467    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-metrics-certs" pod "54ccdd75-9a27-4c5a-9a0a-9277ec782195" (UID: "54ccdd75-9a27-4c5a-9a0a-9277ec782195"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:04.858608 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.858537    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vgvj\" (UniqueName: \"kubernetes.io/projected/54ccdd75-9a27-4c5a-9a0a-9277ec782195-kube-api-access-5vgvj\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:04.858608 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.858563    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:04.858608 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.858573    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ccdd75-9a27-4c5a-9a0a-9277ec782195-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:04.863095 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.863062    5408 generic.go:356] "Generic (PLEG): container finished" podID="54ccdd75-9a27-4c5a-9a0a-9277ec782195" containerID="ea64e23030a1ddc74df8537e77dd4c558b37f11288132e0e01af17a2bc2413d3" exitCode=0
Oct 30 20:02:04.863245 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.863116    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5" event={"ID":"54ccdd75-9a27-4c5a-9a0a-9277ec782195","Type":"ContainerDied","Data":"ea64e23030a1ddc74df8537e77dd4c558b37f11288132e0e01af17a2bc2413d3"}
Oct 30 20:02:04.863245 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.863169    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5" event={"ID":"54ccdd75-9a27-4c5a-9a0a-9277ec782195","Type":"ContainerDied","Data":"0775b93b57966c2690cd8f53b67f4456e519611fdd67b4782030f1f76ad34c60"}
Oct 30 20:02:04.863245 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.863187    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"
Oct 30 20:02:04.863245 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.863197    5408 scope.go:122] "RemoveContainer" containerID="ea64e23030a1ddc74df8537e77dd4c558b37f11288132e0e01af17a2bc2413d3"
Oct 30 20:02:04.916308 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.916273    5408 scope.go:122] "RemoveContainer" containerID="ea64e23030a1ddc74df8537e77dd4c558b37f11288132e0e01af17a2bc2413d3"
Oct 30 20:02:04.933504 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:02:04.933395    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea64e23030a1ddc74df8537e77dd4c558b37f11288132e0e01af17a2bc2413d3\": container with ID starting with ea64e23030a1ddc74df8537e77dd4c558b37f11288132e0e01af17a2bc2413d3 not found: ID does not exist" containerID="ea64e23030a1ddc74df8537e77dd4c558b37f11288132e0e01af17a2bc2413d3"
Oct 30 20:02:04.933504 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.933465    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea64e23030a1ddc74df8537e77dd4c558b37f11288132e0e01af17a2bc2413d3"} err="failed to get container status \"ea64e23030a1ddc74df8537e77dd4c558b37f11288132e0e01af17a2bc2413d3\": rpc error: code = NotFound desc = could not find container \"ea64e23030a1ddc74df8537e77dd4c558b37f11288132e0e01af17a2bc2413d3\": container with ID starting with ea64e23030a1ddc74df8537e77dd4c558b37f11288132e0e01af17a2bc2413d3 not found: ID does not exist"
Oct 30 20:02:04.969275 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.969230    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"]
Oct 30 20:02:04.982915 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:04.982859    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4c072-558d57467-2csf5"]
Oct 30 20:02:05.533371 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:05.533339    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54ccdd75-9a27-4c5a-9a0a-9277ec782195" path="/var/lib/kubelet/pods/54ccdd75-9a27-4c5a-9a0a-9277ec782195/volumes"
Oct 30 20:02:05.533872 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:05.533858    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b8025ef-9bd4-463a-bc56-50d8d46673c2" path="/var/lib/kubelet/pods/8b8025ef-9bd4-463a-bc56-50d8d46673c2/volumes"
Oct 30 20:02:05.534352 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:05.534338    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" path="/var/lib/kubelet/pods/9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec/volumes"
Oct 30 20:02:06.754495 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:06.754464    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"]
Oct 30 20:02:06.754990 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:06.754769    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz" podUID="6c08ea6f-3cde-4826-b83f-1397530f484d" containerName="maas-api" containerID="cri-o://76eefac513db885b8ea7bda31e006ad339606d6a1f8e05f3e6db236404e3bda0" gracePeriod=30
Oct 30 20:02:06.899937 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:06.899897    5408 generic.go:356] "Generic (PLEG): container finished" podID="6c08ea6f-3cde-4826-b83f-1397530f484d" containerID="76eefac513db885b8ea7bda31e006ad339606d6a1f8e05f3e6db236404e3bda0" exitCode=0
Oct 30 20:02:06.900179 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:06.900093    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz" event={"ID":"6c08ea6f-3cde-4826-b83f-1397530f484d","Type":"ContainerDied","Data":"76eefac513db885b8ea7bda31e006ad339606d6a1f8e05f3e6db236404e3bda0"}
Oct 30 20:02:07.547698 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:07.547672    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 20:02:07.747056 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:07.739613    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c08ea6f-3cde-4826-b83f-1397530f484d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c08ea6f-3cde-4826-b83f-1397530f484d-openshift-service-ca\") pod \"6c08ea6f-3cde-4826-b83f-1397530f484d\" (UID: \"6c08ea6f-3cde-4826-b83f-1397530f484d\") "
Oct 30 20:02:07.760061 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:07.747493    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c08ea6f-3cde-4826-b83f-1397530f484d-kube-api-access-rwj4n\" (UniqueName: \"kubernetes.io/projected/6c08ea6f-3cde-4826-b83f-1397530f484d-kube-api-access-rwj4n\") pod \"6c08ea6f-3cde-4826-b83f-1397530f484d\" (UID: \"6c08ea6f-3cde-4826-b83f-1397530f484d\") "
Oct 30 20:02:07.818299 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:07.818265    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c08ea6f-3cde-4826-b83f-1397530f484d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c08ea6f-3cde-4826-b83f-1397530f484d-maas-api-tls\") pod \"6c08ea6f-3cde-4826-b83f-1397530f484d\" (UID: \"6c08ea6f-3cde-4826-b83f-1397530f484d\") "
Oct 30 20:02:07.829160 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:07.829126    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c08ea6f-3cde-4826-b83f-1397530f484d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c08ea6f-3cde-4826-b83f-1397530f484d-metrics-certs\") pod \"6c08ea6f-3cde-4826-b83f-1397530f484d\" (UID: \"6c08ea6f-3cde-4826-b83f-1397530f484d\") "
Oct 30 20:02:07.856790 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:07.786183    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c08ea6f-3cde-4826-b83f-1397530f484d-openshift-service-ca" pod "6c08ea6f-3cde-4826-b83f-1397530f484d" (UID: "6c08ea6f-3cde-4826-b83f-1397530f484d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:07.856790 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:07.847702    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c08ea6f-3cde-4826-b83f-1397530f484d-maas-api-tls" pod "6c08ea6f-3cde-4826-b83f-1397530f484d" (UID: "6c08ea6f-3cde-4826-b83f-1397530f484d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:07.857954 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:07.857926    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c08ea6f-3cde-4826-b83f-1397530f484d-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:07.858367 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:07.858351    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c08ea6f-3cde-4826-b83f-1397530f484d-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:07.858473 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:07.858313    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c08ea6f-3cde-4826-b83f-1397530f484d-kube-api-access-rwj4n" pod "6c08ea6f-3cde-4826-b83f-1397530f484d" (UID: "6c08ea6f-3cde-4826-b83f-1397530f484d"). InnerVolumeSpecName "kube-api-access-rwj4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:07.904053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:07.896683    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c08ea6f-3cde-4826-b83f-1397530f484d-metrics-certs" pod "6c08ea6f-3cde-4826-b83f-1397530f484d" (UID: "6c08ea6f-3cde-4826-b83f-1397530f484d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:07.935097 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:07.923390    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz" event={"ID":"6c08ea6f-3cde-4826-b83f-1397530f484d","Type":"ContainerDied","Data":"2ad7341c57f3da35129565830ae86f38e1199dc9ed816a96db4b5fa4d83f0279"}
Oct 30 20:02:07.935097 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:07.923459    5408 scope.go:122] "RemoveContainer" containerID="76eefac513db885b8ea7bda31e006ad339606d6a1f8e05f3e6db236404e3bda0"
Oct 30 20:02:07.935097 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:07.923603    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"
Oct 30 20:02:07.960278 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:07.960159    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwj4n\" (UniqueName: \"kubernetes.io/projected/6c08ea6f-3cde-4826-b83f-1397530f484d-kube-api-access-rwj4n\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:07.960536 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:07.960514    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c08ea6f-3cde-4826-b83f-1397530f484d-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:07.978547 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:07.978298    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"]
Oct 30 20:02:07.984136 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:07.984103    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd1b04-54bf8889b9-hlhgz"]
Oct 30 20:02:09.574057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:09.573406    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c08ea6f-3cde-4826-b83f-1397530f484d" path="/var/lib/kubelet/pods/6c08ea6f-3cde-4826-b83f-1397530f484d/volumes"
Oct 30 20:02:09.759052 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:02:09.749346    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:02:14.941340 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.941299    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"]
Oct 30 20:02:14.942797 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.942768    5408 cpu_manager.go:414] "containerMap: removing container" podUID="6c08ea6f-3cde-4826-b83f-1397530f484d" containerName="maas-api"
Oct 30 20:02:14.942960 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.942944    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c08ea6f-3cde-4826-b83f-1397530f484d" containerName="maas-api"
Oct 30 20:02:14.943084 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.943058    5408 cpu_manager.go:414] "containerMap: removing container" podUID="54ccdd75-9a27-4c5a-9a0a-9277ec782195" containerName="maas-api"
Oct 30 20:02:14.943084 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.943076    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ccdd75-9a27-4c5a-9a0a-9277ec782195" containerName="maas-api"
Oct 30 20:02:14.943278 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.943094    5408 cpu_manager.go:414] "containerMap: removing container" podUID="8b8025ef-9bd4-463a-bc56-50d8d46673c2" containerName="istio-proxy"
Oct 30 20:02:14.943278 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.943103    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b8025ef-9bd4-463a-bc56-50d8d46673c2" containerName="istio-proxy"
Oct 30 20:02:14.943278 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.943134    5408 cpu_manager.go:414] "containerMap: removing container" podUID="2d3a55c8-98fe-4911-b3c5-476a14bc809a" containerName="istio-proxy"
Oct 30 20:02:14.943278 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.943141    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d3a55c8-98fe-4911-b3c5-476a14bc809a" containerName="istio-proxy"
Oct 30 20:02:14.943278 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.943163    5408 cpu_manager.go:414] "containerMap: removing container" podUID="9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" containerName="istio-proxy"
Oct 30 20:02:14.943278 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.943170    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" containerName="istio-proxy"
Oct 30 20:02:14.943278 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.943205    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f96b2a7d-14e7-4386-921d-b3bb1d6ec55d" containerName="main"
Oct 30 20:02:14.943278 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.943212    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f96b2a7d-14e7-4386-921d-b3bb1d6ec55d" containerName="main"
Oct 30 20:02:14.943685 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.943383    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c08ea6f-3cde-4826-b83f-1397530f484d" containerName="maas-api"
Oct 30 20:02:14.943685 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.943400    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ccdd75-9a27-4c5a-9a0a-9277ec782195" containerName="maas-api"
Oct 30 20:02:14.943685 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.943408    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b8025ef-9bd4-463a-bc56-50d8d46673c2" containerName="istio-proxy"
Oct 30 20:02:14.943685 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.943417    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d3a55c8-98fe-4911-b3c5-476a14bc809a" containerName="istio-proxy"
Oct 30 20:02:14.943685 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.943425    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c5fa6c2-e2dc-43d5-bdae-a0bb0c9312ec" containerName="istio-proxy"
Oct 30 20:02:14.943685 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.943436    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="f96b2a7d-14e7-4386-921d-b3bb1d6ec55d" containerName="main"
Oct 30 20:02:14.947918 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.947873    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:14.953133 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.952798    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"]
Oct 30 20:02:14.960906 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:14.960883    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4f244d-openshift-default-dockercfg-n2fdz\""
Oct 30 20:02:15.060081 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.049069    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-podinfo\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.060081 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.049134    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw7b7\" (UniqueName: \"kubernetes.io/projected/e0c23612-28be-4e5f-aa95-71c40e82450e-kube-api-access-vw7b7\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.060081 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.049268    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-workload-socket\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.060081 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.049304    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-credential-socket\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.060081 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.049347    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-data\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.060081 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.049405    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-token\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.060081 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.049438    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0c23612-28be-4e5f-aa95-71c40e82450e-istiod-ca-cert\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.060081 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.049680    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-workload-certs\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.060081 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.049720    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-envoy\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.157053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.153912    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-workload-certs\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.157053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.153981    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-envoy\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.157053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.154102    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-podinfo\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.157053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.154133    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw7b7\" (UniqueName: \"kubernetes.io/projected/e0c23612-28be-4e5f-aa95-71c40e82450e-kube-api-access-vw7b7\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.157053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.154262    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-workload-socket\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.157053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.154288    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-credential-socket\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.157053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.154324    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-data\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.157053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.154380    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-token\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.157053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.154410    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0c23612-28be-4e5f-aa95-71c40e82450e-istiod-ca-cert\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.157053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.155608    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0c23612-28be-4e5f-aa95-71c40e82450e-istiod-ca-cert\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.157053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.155913    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-workload-certs\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.216098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.203177    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-envoy\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.216642 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.216578    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-workload-socket\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.217009 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.216988    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-data\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.217456 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.217431    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-credential-socket\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.269094 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.252642    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-token\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.278053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.270680    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-podinfo\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.278053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.271851    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw7b7\" (UniqueName: \"kubernetes.io/projected/e0c23612-28be-4e5f-aa95-71c40e82450e-kube-api-access-vw7b7\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.283884 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.283637    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:15.888795 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:02:15.888757    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c23612_28be_4e5f_aa95_71c40e82450e.slice/crio-3e980333295940baea2db90b1c2dc8fd01f9b402e65ceb2db86500fa8874fecd WatchSource:0}: Error finding container 3e980333295940baea2db90b1c2dc8fd01f9b402e65ceb2db86500fa8874fecd: Status 404 returned error can't find the container with id 3e980333295940baea2db90b1c2dc8fd01f9b402e65ceb2db86500fa8874fecd
Oct 30 20:02:15.891390 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.891349    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:02:15.891494 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.891436    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:02:15.891494 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.891485    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:02:15.903938 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:15.903903    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"]
Oct 30 20:02:16.057605 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:16.056406    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj" event={"ID":"e0c23612-28be-4e5f-aa95-71c40e82450e","Type":"ContainerStarted","Data":"7a18ecc51e65dc731022f8c617b5d0794e8c7b58b5ca85e8e7962b5e2658d9e0"}
Oct 30 20:02:16.057605 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:16.056466    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj" event={"ID":"e0c23612-28be-4e5f-aa95-71c40e82450e","Type":"ContainerStarted","Data":"3e980333295940baea2db90b1c2dc8fd01f9b402e65ceb2db86500fa8874fecd"}
Oct 30 20:02:16.124054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:16.123783    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj" podStartSLOduration=2.123764128 podStartE2EDuration="2.123764128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:16.120373495 +0000 UTC m=+1890.704643113" watchObservedRunningTime="2026-10-30 20:02:16.123764128 +0000 UTC m=+1890.708033745"
Oct 30 20:02:16.284860 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:16.284828    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:16.286953 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:16.286924    5408 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:02:16.287107 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:16.286983    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj" podUID="e0c23612-28be-4e5f-aa95-71c40e82450e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 20:02:17.293628 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:17.293593    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:18.091842 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:18.091806    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:18.093016 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:18.092985    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:02:18.636514 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:18.636477    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"]
Oct 30 20:02:18.637145 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:18.636724    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f" podUID="a6aa0a29-6f37-4b42-92f1-95707639c58a" containerName="istio-proxy" containerID="cri-o://d7ff1041f51b112c2c23effd347b3d6e35fb6eb39c177d18f104301c0939e91a" gracePeriod=30
Oct 30 20:02:20.967138 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:20.967088    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477"]
Oct 30 20:02:20.973520 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:20.973497    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477"
Oct 30 20:02:21.006971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.006923    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl"]
Oct 30 20:02:21.011267 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.011226    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4f244d\""
Oct 30 20:02:21.011584 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.011562    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4f244d-dockercfg-zhbr9\""
Oct 30 20:02:21.018233 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.018212    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477"]
Oct 30 20:02:21.018368 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.018350    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl"
Oct 30 20:02:21.049942 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.049240    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl"]
Oct 30 20:02:21.131141 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.120421    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7mb5\" (UniqueName: \"kubernetes.io/projected/38fa73eb-88a7-4868-8e33-a407edc23aef-kube-api-access-v7mb5\") pod \"payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477\" (UID: \"38fa73eb-88a7-4868-8e33-a407edc23aef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477"
Oct 30 20:02:21.131141 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.120613    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38fa73eb-88a7-4868-8e33-a407edc23aef-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477\" (UID: \"38fa73eb-88a7-4868-8e33-a407edc23aef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477"
Oct 30 20:02:21.131141 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.120762    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lgzv\" (UniqueName: \"kubernetes.io/projected/c27eb3c2-6159-459c-8ef1-6ef50c9b62cc-kube-api-access-6lgzv\") pod \"payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl\" (UID: \"c27eb3c2-6159-459c-8ef1-6ef50c9b62cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl"
Oct 30 20:02:21.131141 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.120880    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27eb3c2-6159-459c-8ef1-6ef50c9b62cc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl\" (UID: \"c27eb3c2-6159-459c-8ef1-6ef50c9b62cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl"
Oct 30 20:02:21.234098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.230322    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7mb5\" (UniqueName: \"kubernetes.io/projected/38fa73eb-88a7-4868-8e33-a407edc23aef-kube-api-access-v7mb5\") pod \"payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477\" (UID: \"38fa73eb-88a7-4868-8e33-a407edc23aef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477"
Oct 30 20:02:21.234098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.230442    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38fa73eb-88a7-4868-8e33-a407edc23aef-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477\" (UID: \"38fa73eb-88a7-4868-8e33-a407edc23aef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477"
Oct 30 20:02:21.234098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.230592    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lgzv\" (UniqueName: \"kubernetes.io/projected/c27eb3c2-6159-459c-8ef1-6ef50c9b62cc-kube-api-access-6lgzv\") pod \"payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl\" (UID: \"c27eb3c2-6159-459c-8ef1-6ef50c9b62cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl"
Oct 30 20:02:21.234098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.230705    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27eb3c2-6159-459c-8ef1-6ef50c9b62cc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl\" (UID: \"c27eb3c2-6159-459c-8ef1-6ef50c9b62cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl"
Oct 30 20:02:21.234098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.231669    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27eb3c2-6159-459c-8ef1-6ef50c9b62cc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl\" (UID: \"c27eb3c2-6159-459c-8ef1-6ef50c9b62cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl"
Oct 30 20:02:21.234098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.232401    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38fa73eb-88a7-4868-8e33-a407edc23aef-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477\" (UID: \"38fa73eb-88a7-4868-8e33-a407edc23aef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477"
Oct 30 20:02:21.411081 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.405759    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lgzv\" (UniqueName: \"kubernetes.io/projected/c27eb3c2-6159-459c-8ef1-6ef50c9b62cc-kube-api-access-6lgzv\") pod \"payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl\" (UID: \"c27eb3c2-6159-459c-8ef1-6ef50c9b62cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl"
Oct 30 20:02:21.433186 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.426978    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7mb5\" (UniqueName: \"kubernetes.io/projected/38fa73eb-88a7-4868-8e33-a407edc23aef-kube-api-access-v7mb5\") pod \"payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477\" (UID: \"38fa73eb-88a7-4868-8e33-a407edc23aef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477"
Oct 30 20:02:21.589097 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.589052    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477"
Oct 30 20:02:21.639487 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.639459    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl"
Oct 30 20:02:21.909259 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.908829    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477"]
Oct 30 20:02:21.958406 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:21.958372    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl"]
Oct 30 20:02:21.959704 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:02:21.959658    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27eb3c2_6159_459c_8ef1_6ef50c9b62cc.slice/crio-31481a7526bf5903a8d121b49779b6d2089783f03e9add1b5ca51bc44888659c WatchSource:0}: Error finding container 31481a7526bf5903a8d121b49779b6d2089783f03e9add1b5ca51bc44888659c: Status 404 returned error can't find the container with id 31481a7526bf5903a8d121b49779b6d2089783f03e9add1b5ca51bc44888659c
Oct 30 20:02:22.381013 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:22.380976    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl" event={"ID":"c27eb3c2-6159-459c-8ef1-6ef50c9b62cc","Type":"ContainerStarted","Data":"47a3793ee7e943f760f895c3ec1c3176dd4e9efa8034a77c7bfffb002b225c69"}
Oct 30 20:02:22.381577 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:22.381094    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl" event={"ID":"c27eb3c2-6159-459c-8ef1-6ef50c9b62cc","Type":"ContainerStarted","Data":"31481a7526bf5903a8d121b49779b6d2089783f03e9add1b5ca51bc44888659c"}
Oct 30 20:02:22.382595 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:22.382571    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl"
Oct 30 20:02:22.385463 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:22.385438    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477" event={"ID":"38fa73eb-88a7-4868-8e33-a407edc23aef","Type":"ContainerStarted","Data":"c57a296e9ed43b4e4c1ba2e4c1978a5f289545efcaea809a57310cd17b7ca7eb"}
Oct 30 20:02:22.385568 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:22.385493    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477" event={"ID":"38fa73eb-88a7-4868-8e33-a407edc23aef","Type":"ContainerStarted","Data":"4e569b94d6374c61050087b844893aeed3d0dea623d1239c04d8545d8a9e01e8"}
Oct 30 20:02:22.386337 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:22.386317    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477"
Oct 30 20:02:22.440749 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:22.440684    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl" podStartSLOduration=2.440662567 podStartE2EDuration="2.440662567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:22.410524171 +0000 UTC m=+1896.994793788" watchObservedRunningTime="2026-10-30 20:02:22.440662567 +0000 UTC m=+1897.024932184"
Oct 30 20:02:22.497235 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:22.494878    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477" podStartSLOduration=2.494860437 podStartE2EDuration="2.494860437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:22.488798933 +0000 UTC m=+1897.073068546" watchObservedRunningTime="2026-10-30 20:02:22.494860437 +0000 UTC m=+1897.079130055"
Oct 30 20:02:23.395481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:23.395436    5408 generic.go:356] "Generic (PLEG): container finished" podID="138e6bd8-41e4-496d-a8a0-4722f4947a80" containerID="08a66fe5935b35dcc9aa57684afa0dcaba2b4da610edb60f9e1c8038549c5aee" exitCode=0
Oct 30 20:02:23.396070 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:23.395602    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg" event={"ID":"138e6bd8-41e4-496d-a8a0-4722f4947a80","Type":"ContainerDied","Data":"08a66fe5935b35dcc9aa57684afa0dcaba2b4da610edb60f9e1c8038549c5aee"}
Oct 30 20:02:23.396493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:23.396474    5408 scope.go:122] "RemoveContainer" containerID="08a66fe5935b35dcc9aa57684afa0dcaba2b4da610edb60f9e1c8038549c5aee"
Oct 30 20:02:23.398216 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:23.398187    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:02:23.398322 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:23.398257    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:02:23.398322 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:23.398301    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:02:23.990482 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:23.990137    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 20:02:24.084636 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.084602    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-credential-socket\") pod \"a6aa0a29-6f37-4b42-92f1-95707639c58a\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") "
Oct 30 20:02:24.084818 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.084663    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-envoy\") pod \"a6aa0a29-6f37-4b42-92f1-95707639c58a\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") "
Oct 30 20:02:24.084818 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.084704    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-token\" (UniqueName: \"kubernetes.io/projected/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-token\") pod \"a6aa0a29-6f37-4b42-92f1-95707639c58a\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") "
Oct 30 20:02:24.084818 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.084770    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-data\") pod \"a6aa0a29-6f37-4b42-92f1-95707639c58a\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") "
Oct 30 20:02:24.084818 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.084801    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-podinfo\") pod \"a6aa0a29-6f37-4b42-92f1-95707639c58a\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") "
Oct 30 20:02:24.085080 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.084867    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-workload-certs\") pod \"a6aa0a29-6f37-4b42-92f1-95707639c58a\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") "
Oct 30 20:02:24.085080 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.084905    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-workload-socket\") pod \"a6aa0a29-6f37-4b42-92f1-95707639c58a\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") "
Oct 30 20:02:24.085080 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.084932    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-credential-socket" pod "a6aa0a29-6f37-4b42-92f1-95707639c58a" (UID: "a6aa0a29-6f37-4b42-92f1-95707639c58a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.085080 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.085021    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6aa0a29-6f37-4b42-92f1-95707639c58a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6aa0a29-6f37-4b42-92f1-95707639c58a-istiod-ca-cert\") pod \"a6aa0a29-6f37-4b42-92f1-95707639c58a\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") "
Oct 30 20:02:24.085304 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.085191    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6aa0a29-6f37-4b42-92f1-95707639c58a-kube-api-access-4mh27\" (UniqueName: \"kubernetes.io/projected/a6aa0a29-6f37-4b42-92f1-95707639c58a-kube-api-access-4mh27\") pod \"a6aa0a29-6f37-4b42-92f1-95707639c58a\" (UID: \"a6aa0a29-6f37-4b42-92f1-95707639c58a\") "
Oct 30 20:02:24.085747 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.085716    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-workload-socket" pod "a6aa0a29-6f37-4b42-92f1-95707639c58a" (UID: "a6aa0a29-6f37-4b42-92f1-95707639c58a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.086167 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.086143    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.086281 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.086173    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.086344 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.086331    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6aa0a29-6f37-4b42-92f1-95707639c58a-istiod-ca-cert" pod "a6aa0a29-6f37-4b42-92f1-95707639c58a" (UID: "a6aa0a29-6f37-4b42-92f1-95707639c58a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:24.086401 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.086390    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-workload-certs" pod "a6aa0a29-6f37-4b42-92f1-95707639c58a" (UID: "a6aa0a29-6f37-4b42-92f1-95707639c58a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.086514 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.086438    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-data" pod "a6aa0a29-6f37-4b42-92f1-95707639c58a" (UID: "a6aa0a29-6f37-4b42-92f1-95707639c58a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.088643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.087851    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-podinfo" pod "a6aa0a29-6f37-4b42-92f1-95707639c58a" (UID: "a6aa0a29-6f37-4b42-92f1-95707639c58a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:24.090466 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.090443    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-envoy" pod "a6aa0a29-6f37-4b42-92f1-95707639c58a" (UID: "a6aa0a29-6f37-4b42-92f1-95707639c58a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.091219 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.091183    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6aa0a29-6f37-4b42-92f1-95707639c58a-kube-api-access-4mh27" pod "a6aa0a29-6f37-4b42-92f1-95707639c58a" (UID: "a6aa0a29-6f37-4b42-92f1-95707639c58a"). InnerVolumeSpecName "kube-api-access-4mh27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:24.092072 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.092018    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-token" pod "a6aa0a29-6f37-4b42-92f1-95707639c58a" (UID: "a6aa0a29-6f37-4b42-92f1-95707639c58a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:24.187988 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.187958    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6aa0a29-6f37-4b42-92f1-95707639c58a-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.187988 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.187986    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mh27\" (UniqueName: \"kubernetes.io/projected/a6aa0a29-6f37-4b42-92f1-95707639c58a-kube-api-access-4mh27\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.187988 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.187996    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.188227 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.188005    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.188227 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.188014    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.188227 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.188022    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6aa0a29-6f37-4b42-92f1-95707639c58a-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.188227 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.188045    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6aa0a29-6f37-4b42-92f1-95707639c58a-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.406834 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.406795    5408 generic.go:356] "Generic (PLEG): container finished" podID="a6aa0a29-6f37-4b42-92f1-95707639c58a" containerID="d7ff1041f51b112c2c23effd347b3d6e35fb6eb39c177d18f104301c0939e91a" exitCode=0
Oct 30 20:02:24.407363 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.406888    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"
Oct 30 20:02:24.407363 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.406923    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f" event={"ID":"a6aa0a29-6f37-4b42-92f1-95707639c58a","Type":"ContainerDied","Data":"d7ff1041f51b112c2c23effd347b3d6e35fb6eb39c177d18f104301c0939e91a"}
Oct 30 20:02:24.407363 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.407008    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f" event={"ID":"a6aa0a29-6f37-4b42-92f1-95707639c58a","Type":"ContainerDied","Data":"51fd55ee6f40e1433f554eb3fc6f0ad3bc422874387be229862a633c23ff2329"}
Oct 30 20:02:24.407363 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.407093    5408 scope.go:122] "RemoveContainer" containerID="d7ff1041f51b112c2c23effd347b3d6e35fb6eb39c177d18f104301c0939e91a"
Oct 30 20:02:24.410942 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.410914    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg" event={"ID":"138e6bd8-41e4-496d-a8a0-4722f4947a80","Type":"ContainerStarted","Data":"16962aec3d752b060c00510d8bc4ff47a180d197d6b490d5911c17a1397731ab"}
Oct 30 20:02:24.421477 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.421460    5408 scope.go:122] "RemoveContainer" containerID="5a96b2d2ea39ce225c4f3585e4688c75559180b7d079cad0938a4fb0bffcf7d6"
Oct 30 20:02:24.442867 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.442845    5408 scope.go:122] "RemoveContainer" containerID="d7ff1041f51b112c2c23effd347b3d6e35fb6eb39c177d18f104301c0939e91a"
Oct 30 20:02:24.443170 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:02:24.443148    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ff1041f51b112c2c23effd347b3d6e35fb6eb39c177d18f104301c0939e91a\": container with ID starting with d7ff1041f51b112c2c23effd347b3d6e35fb6eb39c177d18f104301c0939e91a not found: ID does not exist" containerID="d7ff1041f51b112c2c23effd347b3d6e35fb6eb39c177d18f104301c0939e91a"
Oct 30 20:02:24.443305 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.443177    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ff1041f51b112c2c23effd347b3d6e35fb6eb39c177d18f104301c0939e91a"} err="failed to get container status \"d7ff1041f51b112c2c23effd347b3d6e35fb6eb39c177d18f104301c0939e91a\": rpc error: code = NotFound desc = could not find container \"d7ff1041f51b112c2c23effd347b3d6e35fb6eb39c177d18f104301c0939e91a\": container with ID starting with d7ff1041f51b112c2c23effd347b3d6e35fb6eb39c177d18f104301c0939e91a not found: ID does not exist"
Oct 30 20:02:24.443305 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.443193    5408 scope.go:122] "RemoveContainer" containerID="5a96b2d2ea39ce225c4f3585e4688c75559180b7d079cad0938a4fb0bffcf7d6"
Oct 30 20:02:24.443457 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:02:24.443439    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a96b2d2ea39ce225c4f3585e4688c75559180b7d079cad0938a4fb0bffcf7d6\": container with ID starting with 5a96b2d2ea39ce225c4f3585e4688c75559180b7d079cad0938a4fb0bffcf7d6 not found: ID does not exist" containerID="5a96b2d2ea39ce225c4f3585e4688c75559180b7d079cad0938a4fb0bffcf7d6"
Oct 30 20:02:24.443496 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.443465    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a96b2d2ea39ce225c4f3585e4688c75559180b7d079cad0938a4fb0bffcf7d6"} err="failed to get container status \"5a96b2d2ea39ce225c4f3585e4688c75559180b7d079cad0938a4fb0bffcf7d6\": rpc error: code = NotFound desc = could not find container \"5a96b2d2ea39ce225c4f3585e4688c75559180b7d079cad0938a4fb0bffcf7d6\": container with ID starting with 5a96b2d2ea39ce225c4f3585e4688c75559180b7d079cad0938a4fb0bffcf7d6 not found: ID does not exist"
Oct 30 20:02:24.540358 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.540303    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"]
Oct 30 20:02:24.546849 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.546823    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-26n6f"]
Oct 30 20:02:24.819052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.809363    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:02:24.828671 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.825150    5408 patch_prober.go:29] interesting pod/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 20:02:24.828671 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:24.825213    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg" podUID="138e6bd8-41e4-496d-a8a0-4722f4947a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 20:02:25.546192 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:25.546157    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6aa0a29-6f37-4b42-92f1-95707639c58a" path="/var/lib/kubelet/pods/a6aa0a29-6f37-4b42-92f1-95707639c58a/volumes"
Oct 30 20:02:25.827213 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:25.827129    5408 patch_prober.go:29] interesting pod/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 20:02:25.827464 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:25.827433    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg" podUID="138e6bd8-41e4-496d-a8a0-4722f4947a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 20:02:25.840986 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:25.840955    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl"]
Oct 30 20:02:25.842172 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:25.842149    5408 cpu_manager.go:414] "containerMap: removing container" podUID="a6aa0a29-6f37-4b42-92f1-95707639c58a" containerName="istio-proxy"
Oct 30 20:02:25.842172 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:25.842171    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6aa0a29-6f37-4b42-92f1-95707639c58a" containerName="istio-proxy"
Oct 30 20:02:25.842435 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:25.842420    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6aa0a29-6f37-4b42-92f1-95707639c58a" containerName="istio-proxy"
Oct 30 20:02:25.847116 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:25.847096    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl"
Oct 30 20:02:25.856493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:25.856469    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl"]
Oct 30 20:02:25.860633 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:25.860085    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3a4f02\""
Oct 30 20:02:25.956325 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:25.956294    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b599d3f2-4e57-4072-b8f9-02e0d8422678-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl\" (UID: \"b599d3f2-4e57-4072-b8f9-02e0d8422678\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl"
Oct 30 20:02:25.956513 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:25.956451    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4246p\" (UniqueName: \"kubernetes.io/projected/b599d3f2-4e57-4072-b8f9-02e0d8422678-kube-api-access-4246p\") pod \"payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl\" (UID: \"b599d3f2-4e57-4072-b8f9-02e0d8422678\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl"
Oct 30 20:02:26.131009 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.130914    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b599d3f2-4e57-4072-b8f9-02e0d8422678-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl\" (UID: \"b599d3f2-4e57-4072-b8f9-02e0d8422678\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl"
Oct 30 20:02:26.131430 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.131400    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4246p\" (UniqueName: \"kubernetes.io/projected/b599d3f2-4e57-4072-b8f9-02e0d8422678-kube-api-access-4246p\") pod \"payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl\" (UID: \"b599d3f2-4e57-4072-b8f9-02e0d8422678\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl"
Oct 30 20:02:26.160052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.133010    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b599d3f2-4e57-4072-b8f9-02e0d8422678-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl\" (UID: \"b599d3f2-4e57-4072-b8f9-02e0d8422678\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl"
Oct 30 20:02:26.226432 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.226390    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4246p\" (UniqueName: \"kubernetes.io/projected/b599d3f2-4e57-4072-b8f9-02e0d8422678-kube-api-access-4246p\") pod \"payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl\" (UID: \"b599d3f2-4e57-4072-b8f9-02e0d8422678\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl"
Oct 30 20:02:26.345066 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.344296    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8"]
Oct 30 20:02:26.381147 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.346455    5408 cpu_manager.go:414] "containerMap: removing container" podUID="a6aa0a29-6f37-4b42-92f1-95707639c58a" containerName="istio-proxy"
Oct 30 20:02:26.385323 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.385122    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6aa0a29-6f37-4b42-92f1-95707639c58a" containerName="istio-proxy"
Oct 30 20:02:26.386288 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.385826    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6aa0a29-6f37-4b42-92f1-95707639c58a" containerName="istio-proxy"
Oct 30 20:02:26.418127 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.412576    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8"
Oct 30 20:02:26.438389 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.438359    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3a4f02-dockercfg-fkpck\""
Oct 30 20:02:26.444435 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.444379    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8"]
Oct 30 20:02:26.470701 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.470664    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl"
Oct 30 20:02:26.619092 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.618640    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415f31e1-c808-4fb0-bb8c-33f928248892-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8\" (UID: \"415f31e1-c808-4fb0-bb8c-33f928248892\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8"
Oct 30 20:02:26.619092 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.618772    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb72k\" (UniqueName: \"kubernetes.io/projected/415f31e1-c808-4fb0-bb8c-33f928248892-kube-api-access-tb72k\") pod \"payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8\" (UID: \"415f31e1-c808-4fb0-bb8c-33f928248892\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8"
Oct 30 20:02:26.730104 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.729079    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415f31e1-c808-4fb0-bb8c-33f928248892-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8\" (UID: \"415f31e1-c808-4fb0-bb8c-33f928248892\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8"
Oct 30 20:02:26.730104 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.729224    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb72k\" (UniqueName: \"kubernetes.io/projected/415f31e1-c808-4fb0-bb8c-33f928248892-kube-api-access-tb72k\") pod \"payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8\" (UID: \"415f31e1-c808-4fb0-bb8c-33f928248892\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8"
Oct 30 20:02:26.738052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.731002    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415f31e1-c808-4fb0-bb8c-33f928248892-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8\" (UID: \"415f31e1-c808-4fb0-bb8c-33f928248892\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8"
Oct 30 20:02:26.772052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.766698    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb72k\" (UniqueName: \"kubernetes.io/projected/415f31e1-c808-4fb0-bb8c-33f928248892-kube-api-access-tb72k\") pod \"payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8\" (UID: \"415f31e1-c808-4fb0-bb8c-33f928248892\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8"
Oct 30 20:02:26.795054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.787318    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8"
Oct 30 20:02:26.841541 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.841496    5408 patch_prober.go:29] interesting pod/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 20:02:26.841803 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:26.841772    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg" podUID="138e6bd8-41e4-496d-a8a0-4722f4947a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 20:02:27.268503 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:27.267246    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl"]
Oct 30 20:02:27.269118 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:02:27.269092    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb599d3f2_4e57_4072_b8f9_02e0d8422678.slice/crio-2e6547f7040bf7f210e0ad6e47d46d6fcbc459eccb2423de1c974061a8e77b0a WatchSource:0}: Error finding container 2e6547f7040bf7f210e0ad6e47d46d6fcbc459eccb2423de1c974061a8e77b0a: Status 404 returned error can't find the container with id 2e6547f7040bf7f210e0ad6e47d46d6fcbc459eccb2423de1c974061a8e77b0a
Oct 30 20:02:27.500789 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:27.500756    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl" event={"ID":"b599d3f2-4e57-4072-b8f9-02e0d8422678","Type":"ContainerStarted","Data":"2e6547f7040bf7f210e0ad6e47d46d6fcbc459eccb2423de1c974061a8e77b0a"}
Oct 30 20:02:27.586023 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:27.585544    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8"]
Oct 30 20:02:27.826115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:27.825378    5408 patch_prober.go:29] interesting pod/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 20:02:27.826115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:27.825432    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg" podUID="138e6bd8-41e4-496d-a8a0-4722f4947a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 20:02:28.532970 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:28.532168    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl" event={"ID":"b599d3f2-4e57-4072-b8f9-02e0d8422678","Type":"ContainerStarted","Data":"74ed8dea250356ad06cfa8ee944cd5f882488476a72f84f503930ce47b79fbb0"}
Oct 30 20:02:28.532970 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:28.532917    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl"
Oct 30 20:02:28.534667 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:28.534647    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8" event={"ID":"415f31e1-c808-4fb0-bb8c-33f928248892","Type":"ContainerStarted","Data":"ab9651b4b3424af3dbdd4181cecbad74df4bcca41bdbfef7686f0e4f08a53ec9"}
Oct 30 20:02:28.534862 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:28.534796    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8" event={"ID":"415f31e1-c808-4fb0-bb8c-33f928248892","Type":"ContainerStarted","Data":"6b76c53434dda726da0caaceda5b54294f9910b6c20b11735d69f15c2337256a"}
Oct 30 20:02:28.535448 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:28.535431    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8"
Oct 30 20:02:28.554679 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:28.554638    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl" podStartSLOduration=3.554625837 podStartE2EDuration="3.554625837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:28.553205506 +0000 UTC m=+1903.137475124" watchObservedRunningTime="2026-10-30 20:02:28.554625837 +0000 UTC m=+1903.138895454"
Oct 30 20:02:28.582005 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:28.581962    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8" podStartSLOduration=3.581950368 podStartE2EDuration="3.581950368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:28.581259471 +0000 UTC m=+1903.165529086" watchObservedRunningTime="2026-10-30 20:02:28.581950368 +0000 UTC m=+1903.166219998"
Oct 30 20:02:28.811549 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:28.811458    5408 patch_prober.go:29] interesting pod/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 20:02:28.811925 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:28.811887    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg" podUID="138e6bd8-41e4-496d-a8a0-4722f4947a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 20:02:29.805493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:29.805453    5408 patch_prober.go:29] interesting pod/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 20:02:29.806128 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:29.806094    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg" podUID="138e6bd8-41e4-496d-a8a0-4722f4947a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 20:02:30.165204 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:30.165119    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:02:30.815492 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:30.815460    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:02:30.816736 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:30.816709    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:02:34.450055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:34.449271    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477"
Oct 30 20:02:34.450565 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:34.450520    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl"
Oct 30 20:02:39.662957 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:02:39.662921    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:02:40.439044 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:40.439001    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h"]
Oct 30 20:02:40.443608 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:40.443588    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h"
Oct 30 20:02:40.466478 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:40.466451    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h"]
Oct 30 20:02:40.468360 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:40.468341    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a7bb6c\""
Oct 30 20:02:40.559949 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:40.559919    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl"
Oct 30 20:02:40.560180 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:40.560069    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8"
Oct 30 20:02:40.653052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:40.634083    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2k7c\" (UniqueName: \"kubernetes.io/projected/195cd4cc-15b4-470c-b139-f0dd93530204-kube-api-access-l2k7c\") pod \"payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h\" (UID: \"195cd4cc-15b4-470c-b139-f0dd93530204\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h"
Oct 30 20:02:40.653052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:40.634177    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195cd4cc-15b4-470c-b139-f0dd93530204-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h\" (UID: \"195cd4cc-15b4-470c-b139-f0dd93530204\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h"
Oct 30 20:02:40.746948 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:40.741730    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2k7c\" (UniqueName: \"kubernetes.io/projected/195cd4cc-15b4-470c-b139-f0dd93530204-kube-api-access-l2k7c\") pod \"payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h\" (UID: \"195cd4cc-15b4-470c-b139-f0dd93530204\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h"
Oct 30 20:02:40.783693 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:40.783656    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195cd4cc-15b4-470c-b139-f0dd93530204-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h\" (UID: \"195cd4cc-15b4-470c-b139-f0dd93530204\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h"
Oct 30 20:02:40.808059 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:40.786739    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195cd4cc-15b4-470c-b139-f0dd93530204-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h\" (UID: \"195cd4cc-15b4-470c-b139-f0dd93530204\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h"
Oct 30 20:02:40.952100 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:40.949578    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2k7c\" (UniqueName: \"kubernetes.io/projected/195cd4cc-15b4-470c-b139-f0dd93530204-kube-api-access-l2k7c\") pod \"payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h\" (UID: \"195cd4cc-15b4-470c-b139-f0dd93530204\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h"
Oct 30 20:02:41.057776 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:41.057682    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h"
Oct 30 20:02:41.592021 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:41.591996    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h"]
Oct 30 20:02:41.595211 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:02:41.595186    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195cd4cc_15b4_470c_b139_f0dd93530204.slice/crio-9450686144a19381ffd71afce2a3aada8dde6a650353319d60d2e8016099662e WatchSource:0}: Error finding container 9450686144a19381ffd71afce2a3aada8dde6a650353319d60d2e8016099662e: Status 404 returned error can't find the container with id 9450686144a19381ffd71afce2a3aada8dde6a650353319d60d2e8016099662e
Oct 30 20:02:42.057461 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:42.057425    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h" event={"ID":"195cd4cc-15b4-470c-b139-f0dd93530204","Type":"ContainerStarted","Data":"31c1c6a4e53e1c84b20156561113800671c01c139f532a768b52779ab5f26a84"}
Oct 30 20:02:42.057900 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:42.057484    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h" event={"ID":"195cd4cc-15b4-470c-b139-f0dd93530204","Type":"ContainerStarted","Data":"9450686144a19381ffd71afce2a3aada8dde6a650353319d60d2e8016099662e"}
Oct 30 20:02:42.057900 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:42.057645    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h"
Oct 30 20:02:42.090818 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:42.090757    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h" podStartSLOduration=2.090738392 podStartE2EDuration="2.090738392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:42.089412539 +0000 UTC m=+1916.673682157" watchObservedRunningTime="2026-10-30 20:02:42.090738392 +0000 UTC m=+1916.675008010"
Oct 30 20:02:53.087449 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:53.087398    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h"
Oct 30 20:02:55.915782 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:55.915741    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"]
Oct 30 20:02:55.916332 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:55.916003    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f" podUID="02181f8c-536e-4aa8-bc50-049a4cf03b45" containerName="maas-api" containerID="cri-o://96f536e70186253e7c3c864e0740a6ac29114338c9139105249c47c1b20923ba" gracePeriod=30
Oct 30 20:02:56.688052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:56.650424    5408 generic.go:356] "Generic (PLEG): container finished" podID="02181f8c-536e-4aa8-bc50-049a4cf03b45" containerID="96f536e70186253e7c3c864e0740a6ac29114338c9139105249c47c1b20923ba" exitCode=0
Oct 30 20:02:56.688052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:56.650557    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f" event={"ID":"02181f8c-536e-4aa8-bc50-049a4cf03b45","Type":"ContainerDied","Data":"96f536e70186253e7c3c864e0740a6ac29114338c9139105249c47c1b20923ba"}
Oct 30 20:02:56.722278 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:56.722243    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"
Oct 30 20:02:56.785378 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:56.785337    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02181f8c-536e-4aa8-bc50-049a4cf03b45-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02181f8c-536e-4aa8-bc50-049a4cf03b45-metrics-certs\") pod \"02181f8c-536e-4aa8-bc50-049a4cf03b45\" (UID: \"02181f8c-536e-4aa8-bc50-049a4cf03b45\") "
Oct 30 20:02:56.785573 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:56.785411    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02181f8c-536e-4aa8-bc50-049a4cf03b45-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02181f8c-536e-4aa8-bc50-049a4cf03b45-maas-api-tls\") pod \"02181f8c-536e-4aa8-bc50-049a4cf03b45\" (UID: \"02181f8c-536e-4aa8-bc50-049a4cf03b45\") "
Oct 30 20:02:56.785709 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:56.785694    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02181f8c-536e-4aa8-bc50-049a4cf03b45-kube-api-access-8kgb4\" (UniqueName: \"kubernetes.io/projected/02181f8c-536e-4aa8-bc50-049a4cf03b45-kube-api-access-8kgb4\") pod \"02181f8c-536e-4aa8-bc50-049a4cf03b45\" (UID: \"02181f8c-536e-4aa8-bc50-049a4cf03b45\") "
Oct 30 20:02:56.785806 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:56.785791    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02181f8c-536e-4aa8-bc50-049a4cf03b45-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02181f8c-536e-4aa8-bc50-049a4cf03b45-openshift-service-ca\") pod \"02181f8c-536e-4aa8-bc50-049a4cf03b45\" (UID: \"02181f8c-536e-4aa8-bc50-049a4cf03b45\") "
Oct 30 20:02:56.786259 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:56.786228    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02181f8c-536e-4aa8-bc50-049a4cf03b45-openshift-service-ca" pod "02181f8c-536e-4aa8-bc50-049a4cf03b45" (UID: "02181f8c-536e-4aa8-bc50-049a4cf03b45"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:56.786490 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:56.786470    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02181f8c-536e-4aa8-bc50-049a4cf03b45-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:56.788323 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:56.788300    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02181f8c-536e-4aa8-bc50-049a4cf03b45-kube-api-access-8kgb4" pod "02181f8c-536e-4aa8-bc50-049a4cf03b45" (UID: "02181f8c-536e-4aa8-bc50-049a4cf03b45"). InnerVolumeSpecName "kube-api-access-8kgb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:56.791652 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:56.791627    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02181f8c-536e-4aa8-bc50-049a4cf03b45-metrics-certs" pod "02181f8c-536e-4aa8-bc50-049a4cf03b45" (UID: "02181f8c-536e-4aa8-bc50-049a4cf03b45"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:56.792872 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:56.792846    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02181f8c-536e-4aa8-bc50-049a4cf03b45-maas-api-tls" pod "02181f8c-536e-4aa8-bc50-049a4cf03b45" (UID: "02181f8c-536e-4aa8-bc50-049a4cf03b45"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:56.888237 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:56.888140    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kgb4\" (UniqueName: \"kubernetes.io/projected/02181f8c-536e-4aa8-bc50-049a4cf03b45-kube-api-access-8kgb4\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:56.888237 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:56.888177    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02181f8c-536e-4aa8-bc50-049a4cf03b45-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:56.888237 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:56.888199    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02181f8c-536e-4aa8-bc50-049a4cf03b45-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:57.491891 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:57.491855    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2"]
Oct 30 20:02:57.504041 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:57.503993    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs"]
Oct 30 20:02:57.661683 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:57.661655    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"
Oct 30 20:02:57.661890 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:57.661684    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f" event={"ID":"02181f8c-536e-4aa8-bc50-049a4cf03b45","Type":"ContainerDied","Data":"dd7d17d301fb0b710a24ae323570371b7f10b1663d49c0fb867a88a633e65397"}
Oct 30 20:02:57.661890 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:57.661754    5408 scope.go:122] "RemoveContainer" containerID="96f536e70186253e7c3c864e0740a6ac29114338c9139105249c47c1b20923ba"
Oct 30 20:02:57.691876 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:57.691845    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"]
Oct 30 20:02:57.698971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:57.698945    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c50e1e-84455d4b4d-gbz9f"]
Oct 30 20:02:58.515694 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.515663    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"]
Oct 30 20:02:58.516655 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.516631    5408 cpu_manager.go:414] "containerMap: removing container" podUID="02181f8c-536e-4aa8-bc50-049a4cf03b45" containerName="maas-api"
Oct 30 20:02:58.516779 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.516655    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02181f8c-536e-4aa8-bc50-049a4cf03b45" containerName="maas-api"
Oct 30 20:02:58.516952 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.516934    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="02181f8c-536e-4aa8-bc50-049a4cf03b45" containerName="maas-api"
Oct 30 20:02:58.522161 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.522134    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.529261 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.529225    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-db02a7-openshift-default-dockercfg-tstcx\""
Oct 30 20:02:58.551888 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.551834    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"]
Oct 30 20:02:58.642645 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.642600    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-workload-certs\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.643181 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.643155    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-credential-socket\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.643405 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.643388    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99c11d2-540e-434a-8271-92ea41a6b437-istio-token\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.643604 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.643584    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-istio-data\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.664208 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.664171    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqjwl\" (UniqueName: \"kubernetes.io/projected/f99c11d2-540e-434a-8271-92ea41a6b437-kube-api-access-xqjwl\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.664590 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.664568    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-istio-envoy\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.664733 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.664713    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-workload-socket\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.664853 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.664838    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99c11d2-540e-434a-8271-92ea41a6b437-istio-podinfo\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.664988 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.664973    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99c11d2-540e-434a-8271-92ea41a6b437-istiod-ca-cert\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.789643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.776178    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-workload-certs\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.789643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.776246    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-credential-socket\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.789643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.776295    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99c11d2-540e-434a-8271-92ea41a6b437-istio-token\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.789643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.776325    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-istio-data\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.789643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.776451    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqjwl\" (UniqueName: \"kubernetes.io/projected/f99c11d2-540e-434a-8271-92ea41a6b437-kube-api-access-xqjwl\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.789643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.776579    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-istio-envoy\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.789643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.776614    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-workload-socket\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.789643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.776643    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99c11d2-540e-434a-8271-92ea41a6b437-istio-podinfo\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.789643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.776692    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99c11d2-540e-434a-8271-92ea41a6b437-istiod-ca-cert\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.789643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.777676    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99c11d2-540e-434a-8271-92ea41a6b437-istiod-ca-cert\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.789643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.777954    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-workload-certs\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.789643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.778202    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-credential-socket\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.789643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.778763    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-istio-data\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.789643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.782376    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-workload-socket\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.821784 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.820261    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99c11d2-540e-434a-8271-92ea41a6b437-istio-token\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.825874 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.825840    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99c11d2-540e-434a-8271-92ea41a6b437-istio-podinfo\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.826463 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.826412    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-istio-envoy\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.844872 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.831659    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqjwl\" (UniqueName: \"kubernetes.io/projected/f99c11d2-540e-434a-8271-92ea41a6b437-kube-api-access-xqjwl\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:58.844872 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:58.842317    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:59.504624 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:59.504594    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"]
Oct 30 20:02:59.505539 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:02:59.505506    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99c11d2_540e_434a_8271_92ea41a6b437.slice/crio-836ea83c0c3401ab9a4f6d6ae24cf8276927c97f1b34121195932bb5d7a4623a WatchSource:0}: Error finding container 836ea83c0c3401ab9a4f6d6ae24cf8276927c97f1b34121195932bb5d7a4623a: Status 404 returned error can't find the container with id 836ea83c0c3401ab9a4f6d6ae24cf8276927c97f1b34121195932bb5d7a4623a
Oct 30 20:02:59.509199 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:59.509157    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:02:59.509347 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:59.509246    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:02:59.509347 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:59.509296    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:02:59.539544 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:59.539515    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02181f8c-536e-4aa8-bc50-049a4cf03b45" path="/var/lib/kubelet/pods/02181f8c-536e-4aa8-bc50-049a4cf03b45/volumes"
Oct 30 20:02:59.824479 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:59.824391    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg" event={"ID":"f99c11d2-540e-434a-8271-92ea41a6b437","Type":"ContainerStarted","Data":"f0ec32cfcaf47fd6034c05d17dbafedead82651b9fa951537c0397927bab4d32"}
Oct 30 20:02:59.824479 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:59.824459    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg" event={"ID":"f99c11d2-540e-434a-8271-92ea41a6b437","Type":"ContainerStarted","Data":"836ea83c0c3401ab9a4f6d6ae24cf8276927c97f1b34121195932bb5d7a4623a"}
Oct 30 20:02:59.844472 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:59.844427    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:02:59.862448 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:02:59.862381    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg" podStartSLOduration=1.862361081 podStartE2EDuration="1.862361081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:59.856429292 +0000 UTC m=+1934.440698910" watchObservedRunningTime="2026-10-30 20:02:59.862361081 +0000 UTC m=+1934.446630701"
Oct 30 20:03:00.871454 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:00.869565    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:03:00.873218 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:00.873193    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:03:00.878935 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:00.876515    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:03:02.492893 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.492806    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2" podUID="ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b" containerName="payload-processing" containerID="cri-o://1df72a5c76e29538ace0fb80dcca9f44031a3f84e4e20e865465340fbe6e0a39" gracePeriod=25
Oct 30 20:03:02.505113 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.505084    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs" podUID="4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f" containerName="payload-pre-processing" containerID="cri-o://80f2d94c704d44451b94ae9540ad792820012081e7afa32521cd64681193d32a" gracePeriod=25
Oct 30 20:03:02.810606 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.810578    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2"
Oct 30 20:03:02.834872 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.834766    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs"
Oct 30 20:03:02.880591 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.880556    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f-plugins-config-volume\") pod \"4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f\" (UID: \"4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f\") "
Oct 30 20:03:02.880791 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.880675    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f-kube-api-access-rkjgm\" (UniqueName: \"kubernetes.io/projected/4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f-kube-api-access-rkjgm\") pod \"4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f\" (UID: \"4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f\") "
Oct 30 20:03:02.880876 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.880843    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b-kube-api-access-k4sqs\" (UniqueName: \"kubernetes.io/projected/ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b-kube-api-access-k4sqs\") pod \"ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b\" (UID: \"ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b\") "
Oct 30 20:03:02.880979 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.880961    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b-plugins-config-volume\") pod \"ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b\" (UID: \"ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b\") "
Oct 30 20:03:02.881070 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.881014    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f-plugins-config-volume" pod "4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f" (UID: "4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:02.881617 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.881577    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b-plugins-config-volume" pod "ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b" (UID: "ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:02.881817 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.881792    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:02.881935 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.881821    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:02.883615 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.883593    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f-kube-api-access-rkjgm" pod "4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f" (UID: "4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f"). InnerVolumeSpecName "kube-api-access-rkjgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:02.883728 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.883627    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b-kube-api-access-k4sqs" pod "ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b" (UID: "ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b"). InnerVolumeSpecName "kube-api-access-k4sqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:02.908080 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.907984    5408 generic.go:356] "Generic (PLEG): container finished" podID="4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f" containerID="80f2d94c704d44451b94ae9540ad792820012081e7afa32521cd64681193d32a" exitCode=0
Oct 30 20:03:02.908273 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.908082    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs"
Oct 30 20:03:02.908273 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.908090    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs" event={"ID":"4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f","Type":"ContainerDied","Data":"80f2d94c704d44451b94ae9540ad792820012081e7afa32521cd64681193d32a"}
Oct 30 20:03:02.908273 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.908168    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs" event={"ID":"4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f","Type":"ContainerDied","Data":"9b492aa5cb9ef22495d004228fbce436b61c0ef2c1428dde676f9f8ce9761fdd"}
Oct 30 20:03:02.908273 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.908200    5408 scope.go:122] "RemoveContainer" containerID="80f2d94c704d44451b94ae9540ad792820012081e7afa32521cd64681193d32a"
Oct 30 20:03:02.911299 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.911016    5408 generic.go:356] "Generic (PLEG): container finished" podID="ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b" containerID="1df72a5c76e29538ace0fb80dcca9f44031a3f84e4e20e865465340fbe6e0a39" exitCode=0
Oct 30 20:03:02.911299 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.911140    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2"
Oct 30 20:03:02.911299 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.911107    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2" event={"ID":"ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b","Type":"ContainerDied","Data":"1df72a5c76e29538ace0fb80dcca9f44031a3f84e4e20e865465340fbe6e0a39"}
Oct 30 20:03:02.911299 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.911264    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2" event={"ID":"ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b","Type":"ContainerDied","Data":"3a963a47348996b1f51451dd8735b2353b2e19bf7638f152e82c2b541d0d1629"}
Oct 30 20:03:02.923939 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.923922    5408 scope.go:122] "RemoveContainer" containerID="80f2d94c704d44451b94ae9540ad792820012081e7afa32521cd64681193d32a"
Oct 30 20:03:02.924240 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:03:02.924205    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f2d94c704d44451b94ae9540ad792820012081e7afa32521cd64681193d32a\": container with ID starting with 80f2d94c704d44451b94ae9540ad792820012081e7afa32521cd64681193d32a not found: ID does not exist" containerID="80f2d94c704d44451b94ae9540ad792820012081e7afa32521cd64681193d32a"
Oct 30 20:03:02.924316 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.924245    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f2d94c704d44451b94ae9540ad792820012081e7afa32521cd64681193d32a"} err="failed to get container status \"80f2d94c704d44451b94ae9540ad792820012081e7afa32521cd64681193d32a\": rpc error: code = NotFound desc = could not find container \"80f2d94c704d44451b94ae9540ad792820012081e7afa32521cd64681193d32a\": container with ID starting with 80f2d94c704d44451b94ae9540ad792820012081e7afa32521cd64681193d32a not found: ID does not exist"
Oct 30 20:03:02.924316 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.924261    5408 scope.go:122] "RemoveContainer" containerID="1df72a5c76e29538ace0fb80dcca9f44031a3f84e4e20e865465340fbe6e0a39"
Oct 30 20:03:02.934323 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.934299    5408 scope.go:122] "RemoveContainer" containerID="1df72a5c76e29538ace0fb80dcca9f44031a3f84e4e20e865465340fbe6e0a39"
Oct 30 20:03:02.934594 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:03:02.934574    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df72a5c76e29538ace0fb80dcca9f44031a3f84e4e20e865465340fbe6e0a39\": container with ID starting with 1df72a5c76e29538ace0fb80dcca9f44031a3f84e4e20e865465340fbe6e0a39 not found: ID does not exist" containerID="1df72a5c76e29538ace0fb80dcca9f44031a3f84e4e20e865465340fbe6e0a39"
Oct 30 20:03:02.934665 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.934600    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df72a5c76e29538ace0fb80dcca9f44031a3f84e4e20e865465340fbe6e0a39"} err="failed to get container status \"1df72a5c76e29538ace0fb80dcca9f44031a3f84e4e20e865465340fbe6e0a39\": rpc error: code = NotFound desc = could not find container \"1df72a5c76e29538ace0fb80dcca9f44031a3f84e4e20e865465340fbe6e0a39\": container with ID starting with 1df72a5c76e29538ace0fb80dcca9f44031a3f84e4e20e865465340fbe6e0a39 not found: ID does not exist"
Oct 30 20:03:02.941521 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:02.941494    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs"]
Oct 30 20:03:03.034638 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.034542    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkjgm\" (UniqueName: \"kubernetes.io/projected/4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f-kube-api-access-rkjgm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:03.034638 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.034612    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4sqs\" (UniqueName: \"kubernetes.io/projected/ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b-kube-api-access-k4sqs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:03.057652 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.057552    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ed8b50-679cc9954c-mbmgs"]
Oct 30 20:03:03.087194 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.087161    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2"]
Oct 30 20:03:03.090448 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.090424    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ed8b50-779d6b8448-xdng2"]
Oct 30 20:03:03.538438 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.538405    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f" path="/var/lib/kubelet/pods/4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f/volumes"
Oct 30 20:03:03.538922 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.538904    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b" path="/var/lib/kubelet/pods/ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b/volumes"
Oct 30 20:03:03.621351 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.621323    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p"]
Oct 30 20:03:03.622172 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.622144    5408 cpu_manager.go:414] "containerMap: removing container" podUID="ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b" containerName="payload-processing"
Oct 30 20:03:03.622172 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.622173    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b" containerName="payload-processing"
Oct 30 20:03:03.622344 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.622220    5408 cpu_manager.go:414] "containerMap: removing container" podUID="4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f" containerName="payload-pre-processing"
Oct 30 20:03:03.622344 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.622226    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f" containerName="payload-pre-processing"
Oct 30 20:03:03.622454 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.622361    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed1b2eb5-91e7-4cf8-8bbb-f9eb043b0c1b" containerName="payload-processing"
Oct 30 20:03:03.622454 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.622381    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c52e04b-b1aa-4f44-a01e-0cdef1cdef9f" containerName="payload-pre-processing"
Oct 30 20:03:03.627429 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.627407    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p"
Oct 30 20:03:03.639652 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.639627    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p"]
Oct 30 20:03:03.640840 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.640818    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-db02a7\""
Oct 30 20:03:03.641279 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.641263    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-db02a7-dockercfg-mk7c6\""
Oct 30 20:03:03.656734 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.656709    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2"]
Oct 30 20:03:03.661263 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.661238    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2"
Oct 30 20:03:03.687735 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.687702    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2"]
Oct 30 20:03:03.906104 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.835475    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zhjc\" (UniqueName: \"kubernetes.io/projected/651ba59f-3f1f-479d-805e-30a17661c602-kube-api-access-8zhjc\") pod \"payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p\" (UID: \"651ba59f-3f1f-479d-805e-30a17661c602\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p"
Oct 30 20:03:03.906104 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.835610    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651ba59f-3f1f-479d-805e-30a17661c602-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p\" (UID: \"651ba59f-3f1f-479d-805e-30a17661c602\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p"
Oct 30 20:03:03.906104 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.835681    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aecc895e-a6d4-4fc6-8e55-92b83f17fcde-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2\" (UID: \"aecc895e-a6d4-4fc6-8e55-92b83f17fcde\") " pod="openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2"
Oct 30 20:03:03.906104 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.835772    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmx8l\" (UniqueName: \"kubernetes.io/projected/aecc895e-a6d4-4fc6-8e55-92b83f17fcde-kube-api-access-vmx8l\") pod \"payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2\" (UID: \"aecc895e-a6d4-4fc6-8e55-92b83f17fcde\") " pod="openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2"
Oct 30 20:03:03.956050 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.950096    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zhjc\" (UniqueName: \"kubernetes.io/projected/651ba59f-3f1f-479d-805e-30a17661c602-kube-api-access-8zhjc\") pod \"payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p\" (UID: \"651ba59f-3f1f-479d-805e-30a17661c602\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p"
Oct 30 20:03:03.956050 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.950205    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651ba59f-3f1f-479d-805e-30a17661c602-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p\" (UID: \"651ba59f-3f1f-479d-805e-30a17661c602\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p"
Oct 30 20:03:03.956050 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.950262    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aecc895e-a6d4-4fc6-8e55-92b83f17fcde-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2\" (UID: \"aecc895e-a6d4-4fc6-8e55-92b83f17fcde\") " pod="openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2"
Oct 30 20:03:03.956050 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.950338    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmx8l\" (UniqueName: \"kubernetes.io/projected/aecc895e-a6d4-4fc6-8e55-92b83f17fcde-kube-api-access-vmx8l\") pod \"payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2\" (UID: \"aecc895e-a6d4-4fc6-8e55-92b83f17fcde\") " pod="openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2"
Oct 30 20:03:03.993181 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.983575    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aecc895e-a6d4-4fc6-8e55-92b83f17fcde-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2\" (UID: \"aecc895e-a6d4-4fc6-8e55-92b83f17fcde\") " pod="openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2"
Oct 30 20:03:04.001141 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:03.997487    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651ba59f-3f1f-479d-805e-30a17661c602-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p\" (UID: \"651ba59f-3f1f-479d-805e-30a17661c602\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p"
Oct 30 20:03:04.020056 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:04.019718    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmx8l\" (UniqueName: \"kubernetes.io/projected/aecc895e-a6d4-4fc6-8e55-92b83f17fcde-kube-api-access-vmx8l\") pod \"payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2\" (UID: \"aecc895e-a6d4-4fc6-8e55-92b83f17fcde\") " pod="openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2"
Oct 30 20:03:04.045099 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:04.020431    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zhjc\" (UniqueName: \"kubernetes.io/projected/651ba59f-3f1f-479d-805e-30a17661c602-kube-api-access-8zhjc\") pod \"payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p\" (UID: \"651ba59f-3f1f-479d-805e-30a17661c602\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p"
Oct 30 20:03:04.243101 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:04.243065    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p"
Oct 30 20:03:04.278439 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:04.278399    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2"
Oct 30 20:03:04.635541 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:04.626785    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p"]
Oct 30 20:03:04.681596 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:04.681561    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2"]
Oct 30 20:03:05.061162 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:05.061119    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2" event={"ID":"aecc895e-a6d4-4fc6-8e55-92b83f17fcde","Type":"ContainerStarted","Data":"ad580d4ee05705771ef6145063227f407db1caa8dff11e02721c78abf10fff09"}
Oct 30 20:03:05.061384 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:05.061189    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2" event={"ID":"aecc895e-a6d4-4fc6-8e55-92b83f17fcde","Type":"ContainerStarted","Data":"4eb9c8492ccab967b51c9ca40f381befe4bf916a2cf1d57ec6ee9dbc0f0ecc8b"}
Oct 30 20:03:05.061384 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:05.061221    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2"
Oct 30 20:03:05.063078 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:05.063007    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p" event={"ID":"651ba59f-3f1f-479d-805e-30a17661c602","Type":"ContainerStarted","Data":"8ab54f0292ee3bf38e728dd38f96b66de6b552caf8974b2e129cfa61c7ff08a5"}
Oct 30 20:03:05.063186 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:05.063106    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p" event={"ID":"651ba59f-3f1f-479d-805e-30a17661c602","Type":"ContainerStarted","Data":"6e3b61162081030c52005f7374a9d1df105973f06531935b4ecf258dedb392b9"}
Oct 30 20:03:05.063186 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:05.063138    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p"
Oct 30 20:03:05.086512 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:05.086459    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2" podStartSLOduration=2.086444375 podStartE2EDuration="2.086444375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:05.085000679 +0000 UTC m=+1939.669270299" watchObservedRunningTime="2026-10-30 20:03:05.086444375 +0000 UTC m=+1939.670714019"
Oct 30 20:03:05.117327 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:05.117262    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p" podStartSLOduration=2.117236385 podStartE2EDuration="2.117236385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:05.109582545 +0000 UTC m=+1939.693852163" watchObservedRunningTime="2026-10-30 20:03:05.117236385 +0000 UTC m=+1939.701506005"
Oct 30 20:03:06.882173 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:06.878953    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"]
Oct 30 20:03:06.882173 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:06.879337    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4" podUID="96a3152c-afa9-4f5f-9fb5-ea84606ff044" containerName="istio-proxy" containerID="cri-o://d96e51c5327939728b9ef234c6617463dc4da925980a393fa308d160f9a958eb" gracePeriod=30
Oct 30 20:03:09.662389 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:03:09.662346    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:03:12.207210 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.206951    5408 generic.go:356] "Generic (PLEG): container finished" podID="96a3152c-afa9-4f5f-9fb5-ea84606ff044" containerID="d96e51c5327939728b9ef234c6617463dc4da925980a393fa308d160f9a958eb" exitCode=0
Oct 30 20:03:12.207210 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.207106    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4" event={"ID":"96a3152c-afa9-4f5f-9fb5-ea84606ff044","Type":"ContainerDied","Data":"d96e51c5327939728b9ef234c6617463dc4da925980a393fa308d160f9a958eb"}
Oct 30 20:03:12.288526 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.288496    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:03:12.423875 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.423835    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-credential-socket\") pod \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") "
Oct 30 20:03:12.424069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.423983    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istiod-ca-cert\") pod \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") "
Oct 30 20:03:12.424069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.424010    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-token\" (UniqueName: \"kubernetes.io/projected/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-token\") pod \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") "
Oct 30 20:03:12.424069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.424053    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-workload-socket\") pod \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") "
Oct 30 20:03:12.424204 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.424111    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-podinfo\") pod \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") "
Oct 30 20:03:12.424204 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.424162    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-credential-socket" pod "96a3152c-afa9-4f5f-9fb5-ea84606ff044" (UID: "96a3152c-afa9-4f5f-9fb5-ea84606ff044"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:12.424329 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.424296    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a3152c-afa9-4f5f-9fb5-ea84606ff044-kube-api-access-r2242\" (UniqueName: \"kubernetes.io/projected/96a3152c-afa9-4f5f-9fb5-ea84606ff044-kube-api-access-r2242\") pod \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") "
Oct 30 20:03:12.424393 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.424367    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-workload-socket" pod "96a3152c-afa9-4f5f-9fb5-ea84606ff044" (UID: "96a3152c-afa9-4f5f-9fb5-ea84606ff044"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:12.424455 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.424407    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-data\") pod \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") "
Oct 30 20:03:12.424455 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.424441    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-envoy\") pod \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") "
Oct 30 20:03:12.424560 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.424472    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-workload-certs\") pod \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\" (UID: \"96a3152c-afa9-4f5f-9fb5-ea84606ff044\") "
Oct 30 20:03:12.424560 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.424475    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istiod-ca-cert" pod "96a3152c-afa9-4f5f-9fb5-ea84606ff044" (UID: "96a3152c-afa9-4f5f-9fb5-ea84606ff044"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:12.424697 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.424674    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-data" pod "96a3152c-afa9-4f5f-9fb5-ea84606ff044" (UID: "96a3152c-afa9-4f5f-9fb5-ea84606ff044"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:12.424902 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.424862    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-workload-certs" pod "96a3152c-afa9-4f5f-9fb5-ea84606ff044" (UID: "96a3152c-afa9-4f5f-9fb5-ea84606ff044"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:12.425383 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.425360    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:12.425480 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.425386    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:12.425480 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.425403    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:12.425480 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.425416    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:12.425480 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.425430    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:12.426624 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.426593    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-podinfo" pod "96a3152c-afa9-4f5f-9fb5-ea84606ff044" (UID: "96a3152c-afa9-4f5f-9fb5-ea84606ff044"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:03:12.426816 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.426794    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a3152c-afa9-4f5f-9fb5-ea84606ff044-kube-api-access-r2242" pod "96a3152c-afa9-4f5f-9fb5-ea84606ff044" (UID: "96a3152c-afa9-4f5f-9fb5-ea84606ff044"). InnerVolumeSpecName "kube-api-access-r2242". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:12.426911 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.426838    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-token" pod "96a3152c-afa9-4f5f-9fb5-ea84606ff044" (UID: "96a3152c-afa9-4f5f-9fb5-ea84606ff044"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:12.426959 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.426942    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-envoy" pod "96a3152c-afa9-4f5f-9fb5-ea84606ff044" (UID: "96a3152c-afa9-4f5f-9fb5-ea84606ff044"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:12.527457 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.527421    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:12.527457 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.527458    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:12.527660 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.527474    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2242\" (UniqueName: \"kubernetes.io/projected/96a3152c-afa9-4f5f-9fb5-ea84606ff044-kube-api-access-r2242\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:12.527660 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:12.527490    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a3152c-afa9-4f5f-9fb5-ea84606ff044-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:13.215860 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:13.215822    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4" event={"ID":"96a3152c-afa9-4f5f-9fb5-ea84606ff044","Type":"ContainerDied","Data":"a9b0fa37f889787ae2c177730eb67ac550ea0aaa591ad941ac0785551ab55387"}
Oct 30 20:03:13.216351 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:13.215888    5408 scope.go:122] "RemoveContainer" containerID="d96e51c5327939728b9ef234c6617463dc4da925980a393fa308d160f9a958eb"
Oct 30 20:03:13.216351 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:13.215839    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"
Oct 30 20:03:13.259012 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:13.258873    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"]
Oct 30 20:03:13.263068 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:13.263024    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mwkp4"]
Oct 30 20:03:13.534489 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:13.534455    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a3152c-afa9-4f5f-9fb5-ea84606ff044" path="/var/lib/kubelet/pods/96a3152c-afa9-4f5f-9fb5-ea84606ff044/volumes"
Oct 30 20:03:16.074265 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.074232    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2"
Oct 30 20:03:16.076198 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.076175    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p"
Oct 30 20:03:16.222923 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.222876    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"]
Oct 30 20:03:16.223749 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.223727    5408 cpu_manager.go:414] "containerMap: removing container" podUID="96a3152c-afa9-4f5f-9fb5-ea84606ff044" containerName="istio-proxy"
Oct 30 20:03:16.223911 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.223895    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a3152c-afa9-4f5f-9fb5-ea84606ff044" containerName="istio-proxy"
Oct 30 20:03:16.224282 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.224265    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a3152c-afa9-4f5f-9fb5-ea84606ff044" containerName="istio-proxy"
Oct 30 20:03:16.229798 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.229772    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.245765 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.245729    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-82885f-openshift-default-dockercfg-2ht9s\""
Oct 30 20:03:16.252901 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.252848    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"]
Oct 30 20:03:16.280203 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.280153    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw6fd\" (UniqueName: \"kubernetes.io/projected/50041f27-99a2-47ba-944a-9693459b8c80-kube-api-access-cw6fd\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.280384 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.280244    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50041f27-99a2-47ba-944a-9693459b8c80-istio-token\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.280384 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.280326    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-workload-certs\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.280511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.280388    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50041f27-99a2-47ba-944a-9693459b8c80-istiod-ca-cert\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.280511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.280431    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-workload-socket\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.280603 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.280531    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-istio-envoy\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.280649 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.280615    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50041f27-99a2-47ba-944a-9693459b8c80-istio-podinfo\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.280742 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.280723    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-istio-data\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.280790 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.280774    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-credential-socket\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.414111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.413292    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50041f27-99a2-47ba-944a-9693459b8c80-istio-podinfo\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.414111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.413428    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-istio-data\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.414111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.413462    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-credential-socket\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.414111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.413580    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw6fd\" (UniqueName: \"kubernetes.io/projected/50041f27-99a2-47ba-944a-9693459b8c80-kube-api-access-cw6fd\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.414111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.413634    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50041f27-99a2-47ba-944a-9693459b8c80-istio-token\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.414111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.413718    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-workload-certs\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.414111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.413774    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50041f27-99a2-47ba-944a-9693459b8c80-istiod-ca-cert\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.414111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.413809    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-workload-socket\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.414111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.413919    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-istio-envoy\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.444960 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.444922    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-istio-data\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.452056 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.445463    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-credential-socket\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.452274 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.450999    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50041f27-99a2-47ba-944a-9693459b8c80-istiod-ca-cert\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.478813 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.451262    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-workload-certs\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.480318 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.451487    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-workload-socket\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.596062 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.584755    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw6fd\" (UniqueName: \"kubernetes.io/projected/50041f27-99a2-47ba-944a-9693459b8c80-kube-api-access-cw6fd\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.610054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.596875    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-istio-envoy\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.663063 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.625756    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50041f27-99a2-47ba-944a-9693459b8c80-istio-token\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.863057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.850211    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50041f27-99a2-47ba-944a-9693459b8c80-istio-podinfo\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.866305 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.866270    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:16.910002 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.909969    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z"]
Oct 30 20:03:16.926446 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.926418    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z"
Oct 30 20:03:16.935895 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.935855    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-77b4df-dockercfg-zvrq6\""
Oct 30 20:03:16.936304 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.936278    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-77b4df\""
Oct 30 20:03:16.965698 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:16.965657    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z"]
Oct 30 20:03:17.082052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:17.052440    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b091309-1316-445d-bfee-4ba4ba8b8319-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z\" (UID: \"7b091309-1316-445d-bfee-4ba4ba8b8319\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z"
Oct 30 20:03:17.082052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:17.052528    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdzj5\" (UniqueName: \"kubernetes.io/projected/7b091309-1316-445d-bfee-4ba4ba8b8319-kube-api-access-qdzj5\") pod \"payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z\" (UID: \"7b091309-1316-445d-bfee-4ba4ba8b8319\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z"
Oct 30 20:03:17.177179 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:17.156669    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b091309-1316-445d-bfee-4ba4ba8b8319-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z\" (UID: \"7b091309-1316-445d-bfee-4ba4ba8b8319\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z"
Oct 30 20:03:17.177179 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:17.156761    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdzj5\" (UniqueName: \"kubernetes.io/projected/7b091309-1316-445d-bfee-4ba4ba8b8319-kube-api-access-qdzj5\") pod \"payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z\" (UID: \"7b091309-1316-445d-bfee-4ba4ba8b8319\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z"
Oct 30 20:03:17.177179 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:17.158262    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b091309-1316-445d-bfee-4ba4ba8b8319-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z\" (UID: \"7b091309-1316-445d-bfee-4ba4ba8b8319\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z"
Oct 30 20:03:17.223978 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:17.223908    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdzj5\" (UniqueName: \"kubernetes.io/projected/7b091309-1316-445d-bfee-4ba4ba8b8319-kube-api-access-qdzj5\") pod \"payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z\" (UID: \"7b091309-1316-445d-bfee-4ba4ba8b8319\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z"
Oct 30 20:03:17.307368 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:17.307089    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z"
Oct 30 20:03:17.491537 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:17.491497    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq"]
Oct 30 20:03:17.491732 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:17.491584    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs"]
Oct 30 20:03:17.795063 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:17.785865    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"]
Oct 30 20:03:17.882078 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:17.874744    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:03:17.882078 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:17.874849    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:03:17.882078 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:17.874896    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:03:18.002416 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:17.992777    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z"]
Oct 30 20:03:18.402375 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:18.402346    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z" event={"ID":"7b091309-1316-445d-bfee-4ba4ba8b8319","Type":"ContainerStarted","Data":"bccd8f6848d39c8bc25b3846adb4deda023401192ab274d8a6c0e19e21ec21d5"}
Oct 30 20:03:18.403976 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:18.403952    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v" event={"ID":"50041f27-99a2-47ba-944a-9693459b8c80","Type":"ContainerStarted","Data":"b6c2bc96ec995ca4ca845caeb5762eb5fb2d943b7ba8e8f855551cf6297c1a6e"}
Oct 30 20:03:18.436828 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:18.436770    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v" podStartSLOduration=2.436751768 podStartE2EDuration="2.436751768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:18.431095859 +0000 UTC m=+1953.015365479" watchObservedRunningTime="2026-10-30 20:03:18.436751768 +0000 UTC m=+1953.021021386"
Oct 30 20:03:18.867254 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:18.867217    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:19.457627 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:19.457586    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z" event={"ID":"7b091309-1316-445d-bfee-4ba4ba8b8319","Type":"ContainerStarted","Data":"c2dccb43c476c622ef1bd711eb1053b9504c7e1335bcbbd4aa6e4610e56990ec"}
Oct 30 20:03:19.462134 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:19.462109    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z"
Oct 30 20:03:19.463326 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:19.463288    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v" event={"ID":"50041f27-99a2-47ba-944a-9693459b8c80","Type":"ContainerStarted","Data":"538e6c2c3732c0219e5b2d83dae3cf81bb5381664cece705c695da5c7ae4970b"}
Oct 30 20:03:19.873196 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:19.873160    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:19.935059 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:19.934986    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z" podStartSLOduration=3.934967038 podStartE2EDuration="3.934967038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:19.493696107 +0000 UTC m=+1954.077965723" watchObservedRunningTime="2026-10-30 20:03:19.934967038 +0000 UTC m=+1954.519236655"
Oct 30 20:03:20.501054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:20.496580    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:20.503964 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:20.503934    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:03:22.492449 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:22.492408    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq" podUID="d6458172-491b-41c3-a5a1-d8acc7652807" containerName="payload-processing" containerID="cri-o://fb4ba76c00f95fb985c381dd309f53a30654e6ba8b7ab9b2d007abcf33da5b8f" gracePeriod=25
Oct 30 20:03:22.493875 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:22.492414    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs" podUID="d8634c36-96d5-4ce4-b82f-fd4fd7a6b481" containerName="payload-pre-processing" containerID="cri-o://72fcc56292d3f54cc511729f8ab940f35ffc19f7deee410e1e23aa3cbfc4d001" gracePeriod=25
Oct 30 20:03:23.450196 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.450159    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs"
Oct 30 20:03:23.454912 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.454883    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq"
Oct 30 20:03:23.544858 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.544825    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8634c36-96d5-4ce4-b82f-fd4fd7a6b481-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8634c36-96d5-4ce4-b82f-fd4fd7a6b481-plugins-config-volume\") pod \"d8634c36-96d5-4ce4-b82f-fd4fd7a6b481\" (UID: \"d8634c36-96d5-4ce4-b82f-fd4fd7a6b481\") "
Oct 30 20:03:23.545383 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.544964    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8634c36-96d5-4ce4-b82f-fd4fd7a6b481-kube-api-access-mcdzq\" (UniqueName: \"kubernetes.io/projected/d8634c36-96d5-4ce4-b82f-fd4fd7a6b481-kube-api-access-mcdzq\") pod \"d8634c36-96d5-4ce4-b82f-fd4fd7a6b481\" (UID: \"d8634c36-96d5-4ce4-b82f-fd4fd7a6b481\") "
Oct 30 20:03:23.545383 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.545020    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6458172-491b-41c3-a5a1-d8acc7652807-kube-api-access-chxbv\" (UniqueName: \"kubernetes.io/projected/d6458172-491b-41c3-a5a1-d8acc7652807-kube-api-access-chxbv\") pod \"d6458172-491b-41c3-a5a1-d8acc7652807\" (UID: \"d6458172-491b-41c3-a5a1-d8acc7652807\") "
Oct 30 20:03:23.545383 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.545141    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6458172-491b-41c3-a5a1-d8acc7652807-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6458172-491b-41c3-a5a1-d8acc7652807-plugins-config-volume\") pod \"d6458172-491b-41c3-a5a1-d8acc7652807\" (UID: \"d6458172-491b-41c3-a5a1-d8acc7652807\") "
Oct 30 20:03:23.545555 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.545395    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8634c36-96d5-4ce4-b82f-fd4fd7a6b481-plugins-config-volume" pod "d8634c36-96d5-4ce4-b82f-fd4fd7a6b481" (UID: "d8634c36-96d5-4ce4-b82f-fd4fd7a6b481"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:23.545862 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.545837    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6458172-491b-41c3-a5a1-d8acc7652807-plugins-config-volume" pod "d6458172-491b-41c3-a5a1-d8acc7652807" (UID: "d6458172-491b-41c3-a5a1-d8acc7652807"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:23.546167 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.546140    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6458172-491b-41c3-a5a1-d8acc7652807-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:23.546282 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.546172    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8634c36-96d5-4ce4-b82f-fd4fd7a6b481-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:23.548370 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.548272    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6458172-491b-41c3-a5a1-d8acc7652807-kube-api-access-chxbv" pod "d6458172-491b-41c3-a5a1-d8acc7652807" (UID: "d6458172-491b-41c3-a5a1-d8acc7652807"). InnerVolumeSpecName "kube-api-access-chxbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:23.548492 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.548448    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8634c36-96d5-4ce4-b82f-fd4fd7a6b481-kube-api-access-mcdzq" pod "d8634c36-96d5-4ce4-b82f-fd4fd7a6b481" (UID: "d8634c36-96d5-4ce4-b82f-fd4fd7a6b481"). InnerVolumeSpecName "kube-api-access-mcdzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:23.559598 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.559568    5408 generic.go:356] "Generic (PLEG): container finished" podID="d8634c36-96d5-4ce4-b82f-fd4fd7a6b481" containerID="72fcc56292d3f54cc511729f8ab940f35ffc19f7deee410e1e23aa3cbfc4d001" exitCode=0
Oct 30 20:03:23.559732 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.559671    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs" event={"ID":"d8634c36-96d5-4ce4-b82f-fd4fd7a6b481","Type":"ContainerDied","Data":"72fcc56292d3f54cc511729f8ab940f35ffc19f7deee410e1e23aa3cbfc4d001"}
Oct 30 20:03:23.559732 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.559716    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs" event={"ID":"d8634c36-96d5-4ce4-b82f-fd4fd7a6b481","Type":"ContainerDied","Data":"9c64dd1e272ad86a8609fd5fa777c933b7b6ba344a2a5a8c5ebfa109fb2bc709"}
Oct 30 20:03:23.559856 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.559744    5408 scope.go:122] "RemoveContainer" containerID="72fcc56292d3f54cc511729f8ab940f35ffc19f7deee410e1e23aa3cbfc4d001"
Oct 30 20:03:23.559911 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.559890    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs"
Oct 30 20:03:23.572108 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.570577    5408 generic.go:356] "Generic (PLEG): container finished" podID="d6458172-491b-41c3-a5a1-d8acc7652807" containerID="fb4ba76c00f95fb985c381dd309f53a30654e6ba8b7ab9b2d007abcf33da5b8f" exitCode=0
Oct 30 20:03:23.572108 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.570625    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq" event={"ID":"d6458172-491b-41c3-a5a1-d8acc7652807","Type":"ContainerDied","Data":"fb4ba76c00f95fb985c381dd309f53a30654e6ba8b7ab9b2d007abcf33da5b8f"}
Oct 30 20:03:23.572108 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.570667    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq" event={"ID":"d6458172-491b-41c3-a5a1-d8acc7652807","Type":"ContainerDied","Data":"a3e5755cd94c1ba62dc438541bdc24214f4ad97eb8a035406caf8b009a7c3dd1"}
Oct 30 20:03:23.572108 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.570740    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq"
Oct 30 20:03:23.590943 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.590923    5408 scope.go:122] "RemoveContainer" containerID="72fcc56292d3f54cc511729f8ab940f35ffc19f7deee410e1e23aa3cbfc4d001"
Oct 30 20:03:23.591324 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:03:23.591295    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72fcc56292d3f54cc511729f8ab940f35ffc19f7deee410e1e23aa3cbfc4d001\": container with ID starting with 72fcc56292d3f54cc511729f8ab940f35ffc19f7deee410e1e23aa3cbfc4d001 not found: ID does not exist" containerID="72fcc56292d3f54cc511729f8ab940f35ffc19f7deee410e1e23aa3cbfc4d001"
Oct 30 20:03:23.591422 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.591330    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72fcc56292d3f54cc511729f8ab940f35ffc19f7deee410e1e23aa3cbfc4d001"} err="failed to get container status \"72fcc56292d3f54cc511729f8ab940f35ffc19f7deee410e1e23aa3cbfc4d001\": rpc error: code = NotFound desc = could not find container \"72fcc56292d3f54cc511729f8ab940f35ffc19f7deee410e1e23aa3cbfc4d001\": container with ID starting with 72fcc56292d3f54cc511729f8ab940f35ffc19f7deee410e1e23aa3cbfc4d001 not found: ID does not exist"
Oct 30 20:03:23.591422 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.591348    5408 scope.go:122] "RemoveContainer" containerID="fb4ba76c00f95fb985c381dd309f53a30654e6ba8b7ab9b2d007abcf33da5b8f"
Oct 30 20:03:23.640250 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.637628    5408 scope.go:122] "RemoveContainer" containerID="fb4ba76c00f95fb985c381dd309f53a30654e6ba8b7ab9b2d007abcf33da5b8f"
Oct 30 20:03:23.641474 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:03:23.641007    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4ba76c00f95fb985c381dd309f53a30654e6ba8b7ab9b2d007abcf33da5b8f\": container with ID starting with fb4ba76c00f95fb985c381dd309f53a30654e6ba8b7ab9b2d007abcf33da5b8f not found: ID does not exist" containerID="fb4ba76c00f95fb985c381dd309f53a30654e6ba8b7ab9b2d007abcf33da5b8f"
Oct 30 20:03:23.641474 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.641342    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4ba76c00f95fb985c381dd309f53a30654e6ba8b7ab9b2d007abcf33da5b8f"} err="failed to get container status \"fb4ba76c00f95fb985c381dd309f53a30654e6ba8b7ab9b2d007abcf33da5b8f\": rpc error: code = NotFound desc = could not find container \"fb4ba76c00f95fb985c381dd309f53a30654e6ba8b7ab9b2d007abcf33da5b8f\": container with ID starting with fb4ba76c00f95fb985c381dd309f53a30654e6ba8b7ab9b2d007abcf33da5b8f not found: ID does not exist"
Oct 30 20:03:23.656794 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.656731    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq"]
Oct 30 20:03:23.659771 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.659742    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcdzq\" (UniqueName: \"kubernetes.io/projected/d8634c36-96d5-4ce4-b82f-fd4fd7a6b481-kube-api-access-mcdzq\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:23.659906 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.659891    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chxbv\" (UniqueName: \"kubernetes.io/projected/d6458172-491b-41c3-a5a1-d8acc7652807-kube-api-access-chxbv\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:23.667660 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.667631    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-96ec93-98449bf46-tpgmq"]
Oct 30 20:03:23.671536 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.671516    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs"]
Oct 30 20:03:23.693757 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:23.693733    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-96ec93-5d68655c47-kwgcs"]
Oct 30 20:03:25.555250 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:25.555221    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6458172-491b-41c3-a5a1-d8acc7652807" path="/var/lib/kubelet/pods/d6458172-491b-41c3-a5a1-d8acc7652807/volumes"
Oct 30 20:03:25.555752 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:25.555737    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8634c36-96d5-4ce4-b82f-fd4fd7a6b481" path="/var/lib/kubelet/pods/d8634c36-96d5-4ce4-b82f-fd4fd7a6b481/volumes"
Oct 30 20:03:31.503287 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:31.503258    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z"
Oct 30 20:03:31.760884 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:31.757255    5408 generic.go:356] "Generic (PLEG): container finished" podID="e0c23612-28be-4e5f-aa95-71c40e82450e" containerID="7a18ecc51e65dc731022f8c617b5d0794e8c7b58b5ca85e8e7962b5e2658d9e0" exitCode=0
Oct 30 20:03:31.760884 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:31.757489    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj" event={"ID":"e0c23612-28be-4e5f-aa95-71c40e82450e","Type":"ContainerDied","Data":"7a18ecc51e65dc731022f8c617b5d0794e8c7b58b5ca85e8e7962b5e2658d9e0"}
Oct 30 20:03:31.760884 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:31.758011    5408 scope.go:122] "RemoveContainer" containerID="7a18ecc51e65dc731022f8c617b5d0794e8c7b58b5ca85e8e7962b5e2658d9e0"
Oct 30 20:03:31.761685 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:31.760884    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:03:31.761685 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:31.760982    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:03:31.761685 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:31.761113    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:03:32.766768 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:32.766729    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj" event={"ID":"e0c23612-28be-4e5f-aa95-71c40e82450e","Type":"ContainerStarted","Data":"08f8819f856fcff97a48ed9f69499641fc8669ca4125379ee6e269bdae2f3ac0"}
Oct 30 20:03:33.092489 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:33.092410    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:03:33.285191 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:33.285153    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:03:33.286607 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:33.286582    5408 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:03:33.286731 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:33.286636    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj" podUID="e0c23612-28be-4e5f-aa95-71c40e82450e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 20:03:34.284901 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:34.284864    5408 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:03:34.285379 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:34.284920    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj" podUID="e0c23612-28be-4e5f-aa95-71c40e82450e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 20:03:35.284733 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:35.284694    5408 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:03:35.284983 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:35.284756    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj" podUID="e0c23612-28be-4e5f-aa95-71c40e82450e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 20:03:36.289056 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:36.288398    5408 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:03:36.289056 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:36.288460    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj" podUID="e0c23612-28be-4e5f-aa95-71c40e82450e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 20:03:37.285059 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:37.285007    5408 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:03:37.285348 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:37.285321    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj" podUID="e0c23612-28be-4e5f-aa95-71c40e82450e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 20:03:38.284760 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:38.284730    5408 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:03:38.285187 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:38.284781    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj" podUID="e0c23612-28be-4e5f-aa95-71c40e82450e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 20:03:39.289777 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:39.289749    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:03:39.290792 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:39.290769    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:03:39.722166 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:39.722085    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"]
Oct 30 20:03:39.736750 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:39.731309    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg" podUID="138e6bd8-41e4-496d-a8a0-4722f4947a80" containerName="istio-proxy" containerID="cri-o://16962aec3d752b060c00510d8bc4ff47a180d197d6b490d5911c17a1397731ab" gracePeriod=30
Oct 30 20:03:40.180489 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:03:40.180460    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:03:42.189678 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.189651    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l"]
Oct 30 20:03:42.190394 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.190379    5408 cpu_manager.go:414] "containerMap: removing container" podUID="d6458172-491b-41c3-a5a1-d8acc7652807" containerName="payload-processing"
Oct 30 20:03:42.190456 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.190395    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6458172-491b-41c3-a5a1-d8acc7652807" containerName="payload-processing"
Oct 30 20:03:42.190456 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.190428    5408 cpu_manager.go:414] "containerMap: removing container" podUID="d8634c36-96d5-4ce4-b82f-fd4fd7a6b481" containerName="payload-pre-processing"
Oct 30 20:03:42.190456 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.190434    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8634c36-96d5-4ce4-b82f-fd4fd7a6b481" containerName="payload-pre-processing"
Oct 30 20:03:42.190556 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.190551    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8634c36-96d5-4ce4-b82f-fd4fd7a6b481" containerName="payload-pre-processing"
Oct 30 20:03:42.190589 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.190567    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6458172-491b-41c3-a5a1-d8acc7652807" containerName="payload-processing"
Oct 30 20:03:42.194703 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.194688    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l"
Oct 30 20:03:42.209681 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.209655    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l"]
Oct 30 20:03:42.223388 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.223360    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7"]
Oct 30 20:03:42.225272 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.225239    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d7a733\""
Oct 30 20:03:42.225903 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.225881    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d7a733-dockercfg-gm4j9\""
Oct 30 20:03:42.229306 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.229290    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7"
Oct 30 20:03:42.412057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.399825    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl9dx\" (UniqueName: \"kubernetes.io/projected/e48c3cc5-30d3-4bc8-98a7-548d78595ac6-kube-api-access-cl9dx\") pod \"payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l\" (UID: \"e48c3cc5-30d3-4bc8-98a7-548d78595ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l"
Oct 30 20:03:42.412057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.400174    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48c3cc5-30d3-4bc8-98a7-548d78595ac6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l\" (UID: \"e48c3cc5-30d3-4bc8-98a7-548d78595ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l"
Oct 30 20:03:42.511057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.502309    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl9dx\" (UniqueName: \"kubernetes.io/projected/e48c3cc5-30d3-4bc8-98a7-548d78595ac6-kube-api-access-cl9dx\") pod \"payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l\" (UID: \"e48c3cc5-30d3-4bc8-98a7-548d78595ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l"
Oct 30 20:03:42.511057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.502572    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ggzq\" (UniqueName: \"kubernetes.io/projected/cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1-kube-api-access-5ggzq\") pod \"payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7\" (UID: \"cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7"
Oct 30 20:03:42.511057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.502680    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7\" (UID: \"cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7"
Oct 30 20:03:42.511057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.502723    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48c3cc5-30d3-4bc8-98a7-548d78595ac6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l\" (UID: \"e48c3cc5-30d3-4bc8-98a7-548d78595ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l"
Oct 30 20:03:42.511057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.503642    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48c3cc5-30d3-4bc8-98a7-548d78595ac6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l\" (UID: \"e48c3cc5-30d3-4bc8-98a7-548d78595ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l"
Oct 30 20:03:42.547116 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.547080    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7"]
Oct 30 20:03:42.582052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.571809    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl9dx\" (UniqueName: \"kubernetes.io/projected/e48c3cc5-30d3-4bc8-98a7-548d78595ac6-kube-api-access-cl9dx\") pod \"payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l\" (UID: \"e48c3cc5-30d3-4bc8-98a7-548d78595ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l"
Oct 30 20:03:42.681429 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.681388    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ggzq\" (UniqueName: \"kubernetes.io/projected/cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1-kube-api-access-5ggzq\") pod \"payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7\" (UID: \"cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7"
Oct 30 20:03:42.681776 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.681750    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7\" (UID: \"cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7"
Oct 30 20:03:42.697095 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.683219    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7\" (UID: \"cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7"
Oct 30 20:03:42.737072 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.737015    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ggzq\" (UniqueName: \"kubernetes.io/projected/cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1-kube-api-access-5ggzq\") pod \"payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7\" (UID: \"cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7"
Oct 30 20:03:42.811155 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.811066    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l"
Oct 30 20:03:42.845712 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:42.845665    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7"
Oct 30 20:03:43.115140 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:43.115112    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l"]
Oct 30 20:03:43.115985 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:03:43.115961    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48c3cc5_30d3_4bc8_98a7_548d78595ac6.slice/crio-3cbfcbbb429efca202cad16971fb16e5571b9d2ccb46a69b9c267ebab079108d WatchSource:0}: Error finding container 3cbfcbbb429efca202cad16971fb16e5571b9d2ccb46a69b9c267ebab079108d: Status 404 returned error can't find the container with id 3cbfcbbb429efca202cad16971fb16e5571b9d2ccb46a69b9c267ebab079108d
Oct 30 20:03:43.161354 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:43.161329    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7"]
Oct 30 20:03:44.071838 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:44.070817    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7" event={"ID":"cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1","Type":"ContainerStarted","Data":"8ae0505b97d298038d12aef563c5fe18a53e7b37babbdfd95cc1e2e949a41cab"}
Oct 30 20:03:44.071838 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:44.071114    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7" event={"ID":"cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1","Type":"ContainerStarted","Data":"4ad41ba893b59450afdc16d164480a8bacdc011c7c58ce6ce527d7abfc151a26"}
Oct 30 20:03:44.071838 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:44.071289    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7"
Oct 30 20:03:44.077087 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:44.076272    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l" event={"ID":"e48c3cc5-30d3-4bc8-98a7-548d78595ac6","Type":"ContainerStarted","Data":"3ee741d71f14112e10861253b06cc3a19ed55dd2e6b1fe5d22a0ab083c3c4a60"}
Oct 30 20:03:44.077087 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:44.076325    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l" event={"ID":"e48c3cc5-30d3-4bc8-98a7-548d78595ac6","Type":"ContainerStarted","Data":"3cbfcbbb429efca202cad16971fb16e5571b9d2ccb46a69b9c267ebab079108d"}
Oct 30 20:03:44.077314 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:44.077126    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l"
Oct 30 20:03:44.115261 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:44.115212    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7" podStartSLOduration=2.115198038 podStartE2EDuration="2.115198038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:44.111281888 +0000 UTC m=+1978.695551507" watchObservedRunningTime="2026-10-30 20:03:44.115198038 +0000 UTC m=+1978.699467656"
Oct 30 20:03:44.138585 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:44.138528    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l" podStartSLOduration=2.138509335 podStartE2EDuration="2.138509335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:44.133729686 +0000 UTC m=+1978.717999303" watchObservedRunningTime="2026-10-30 20:03:44.138509335 +0000 UTC m=+1978.722778956"
Oct 30 20:03:45.086874 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.086840    5408 generic.go:356] "Generic (PLEG): container finished" podID="138e6bd8-41e4-496d-a8a0-4722f4947a80" containerID="16962aec3d752b060c00510d8bc4ff47a180d197d6b490d5911c17a1397731ab" exitCode=0
Oct 30 20:03:45.087305 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.086876    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg" event={"ID":"138e6bd8-41e4-496d-a8a0-4722f4947a80","Type":"ContainerDied","Data":"16962aec3d752b060c00510d8bc4ff47a180d197d6b490d5911c17a1397731ab"}
Oct 30 20:03:45.087305 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.086939    5408 scope.go:122] "RemoveContainer" containerID="08a66fe5935b35dcc9aa57684afa0dcaba2b4da610edb60f9e1c8038549c5aee"
Oct 30 20:03:45.750977 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.750946    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:03:45.848708 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.848671    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-workload-certs\") pod \"138e6bd8-41e4-496d-a8a0-4722f4947a80\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") "
Oct 30 20:03:45.848899 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.848770    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-credential-socket\") pod \"138e6bd8-41e4-496d-a8a0-4722f4947a80\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") "
Oct 30 20:03:45.848899 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.848829    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-token\" (UniqueName: \"kubernetes.io/projected/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-token\") pod \"138e6bd8-41e4-496d-a8a0-4722f4947a80\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") "
Oct 30 20:03:45.849094 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.848931    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/138e6bd8-41e4-496d-a8a0-4722f4947a80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138e6bd8-41e4-496d-a8a0-4722f4947a80-istiod-ca-cert\") pod \"138e6bd8-41e4-496d-a8a0-4722f4947a80\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") "
Oct 30 20:03:45.849094 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.848976    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-podinfo\") pod \"138e6bd8-41e4-496d-a8a0-4722f4947a80\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") "
Oct 30 20:03:45.849094 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.848974    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-workload-certs" pod "138e6bd8-41e4-496d-a8a0-4722f4947a80" (UID: "138e6bd8-41e4-496d-a8a0-4722f4947a80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:45.849094 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.849074    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-credential-socket" pod "138e6bd8-41e4-496d-a8a0-4722f4947a80" (UID: "138e6bd8-41e4-496d-a8a0-4722f4947a80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:45.849341 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.849188    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-envoy\") pod \"138e6bd8-41e4-496d-a8a0-4722f4947a80\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") "
Oct 30 20:03:45.849341 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.849278    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-data\") pod \"138e6bd8-41e4-496d-a8a0-4722f4947a80\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") "
Oct 30 20:03:45.849341 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.849330    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138e6bd8-41e4-496d-a8a0-4722f4947a80-kube-api-access-tf8nl\" (UniqueName: \"kubernetes.io/projected/138e6bd8-41e4-496d-a8a0-4722f4947a80-kube-api-access-tf8nl\") pod \"138e6bd8-41e4-496d-a8a0-4722f4947a80\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") "
Oct 30 20:03:45.849499 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.849387    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-workload-socket\") pod \"138e6bd8-41e4-496d-a8a0-4722f4947a80\" (UID: \"138e6bd8-41e4-496d-a8a0-4722f4947a80\") "
Oct 30 20:03:45.850057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.849332    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/138e6bd8-41e4-496d-a8a0-4722f4947a80-istiod-ca-cert" pod "138e6bd8-41e4-496d-a8a0-4722f4947a80" (UID: "138e6bd8-41e4-496d-a8a0-4722f4947a80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:45.850057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.849815    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-workload-socket" pod "138e6bd8-41e4-496d-a8a0-4722f4947a80" (UID: "138e6bd8-41e4-496d-a8a0-4722f4947a80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:45.850236 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.850172    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:45.850236 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.850194    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138e6bd8-41e4-496d-a8a0-4722f4947a80-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:45.850236 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.850209    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:45.850236 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.850224    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:45.850541 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.850510    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-data" pod "138e6bd8-41e4-496d-a8a0-4722f4947a80" (UID: "138e6bd8-41e4-496d-a8a0-4722f4947a80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:45.851261 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.851240    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-token" pod "138e6bd8-41e4-496d-a8a0-4722f4947a80" (UID: "138e6bd8-41e4-496d-a8a0-4722f4947a80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:45.851898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.851874    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-envoy" pod "138e6bd8-41e4-496d-a8a0-4722f4947a80" (UID: "138e6bd8-41e4-496d-a8a0-4722f4947a80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:45.852241 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.852215    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-podinfo" pod "138e6bd8-41e4-496d-a8a0-4722f4947a80" (UID: "138e6bd8-41e4-496d-a8a0-4722f4947a80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:03:45.852863 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.852843    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138e6bd8-41e4-496d-a8a0-4722f4947a80-kube-api-access-tf8nl" pod "138e6bd8-41e4-496d-a8a0-4722f4947a80" (UID: "138e6bd8-41e4-496d-a8a0-4722f4947a80"). InnerVolumeSpecName "kube-api-access-tf8nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:45.952215 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.952184    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:45.952215 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.952217    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:45.952422 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.952233    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:45.952422 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.952250    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf8nl\" (UniqueName: \"kubernetes.io/projected/138e6bd8-41e4-496d-a8a0-4722f4947a80-kube-api-access-tf8nl\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:45.952422 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:45.952264    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138e6bd8-41e4-496d-a8a0-4722f4947a80-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:46.096992 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:46.096960    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"
Oct 30 20:03:46.096992 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:46.096982    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg" event={"ID":"138e6bd8-41e4-496d-a8a0-4722f4947a80","Type":"ContainerDied","Data":"be4589f82bef2c48f1ce091b3d59445acecf1fa37b63d3d5be70acbca776f0e8"}
Oct 30 20:03:46.097528 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:46.097066    5408 scope.go:122] "RemoveContainer" containerID="16962aec3d752b060c00510d8bc4ff47a180d197d6b490d5911c17a1397731ab"
Oct 30 20:03:46.180918 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:46.179831    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"]
Oct 30 20:03:46.181162 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:46.181138    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ed8b50-openshift-default-57bbf6ff6c-7szjg"]
Oct 30 20:03:47.362173 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.362140    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"]
Oct 30 20:03:47.363058 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.363040    5408 cpu_manager.go:414] "containerMap: removing container" podUID="138e6bd8-41e4-496d-a8a0-4722f4947a80" containerName="istio-proxy"
Oct 30 20:03:47.363104 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.363059    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="138e6bd8-41e4-496d-a8a0-4722f4947a80" containerName="istio-proxy"
Oct 30 20:03:47.363104 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.363079    5408 cpu_manager.go:414] "containerMap: removing container" podUID="138e6bd8-41e4-496d-a8a0-4722f4947a80" containerName="istio-proxy"
Oct 30 20:03:47.363104 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.363085    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="138e6bd8-41e4-496d-a8a0-4722f4947a80" containerName="istio-proxy"
Oct 30 20:03:47.363204 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.363189    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="138e6bd8-41e4-496d-a8a0-4722f4947a80" containerName="istio-proxy"
Oct 30 20:03:47.363240 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.363206    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="138e6bd8-41e4-496d-a8a0-4722f4947a80" containerName="istio-proxy"
Oct 30 20:03:47.368763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.368740    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"
Oct 30 20:03:47.385349 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.385329    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-82885f-serving-cert\""
Oct 30 20:03:47.386806 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.386781    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"]
Oct 30 20:03:47.502246 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.502209    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-metrics-certs\") pod \"maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc\" (UID: \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"
Oct 30 20:03:47.502635 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.502609    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwvx7\" (UniqueName: \"kubernetes.io/projected/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-kube-api-access-nwvx7\") pod \"maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc\" (UID: \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"
Oct 30 20:03:47.502832 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.502647    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-openshift-service-ca\") pod \"maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc\" (UID: \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"
Oct 30 20:03:47.502832 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.502770    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-maas-api-tls\") pod \"maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc\" (UID: \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"
Oct 30 20:03:47.533904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.533875    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="138e6bd8-41e4-496d-a8a0-4722f4947a80" path="/var/lib/kubelet/pods/138e6bd8-41e4-496d-a8a0-4722f4947a80/volumes"
Oct 30 20:03:47.604148 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.604109    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwvx7\" (UniqueName: \"kubernetes.io/projected/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-kube-api-access-nwvx7\") pod \"maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc\" (UID: \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"
Oct 30 20:03:47.604329 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.604157    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-openshift-service-ca\") pod \"maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc\" (UID: \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"
Oct 30 20:03:47.604329 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.604188    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-maas-api-tls\") pod \"maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc\" (UID: \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"
Oct 30 20:03:47.604535 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.604517    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-metrics-certs\") pod \"maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc\" (UID: \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"
Oct 30 20:03:47.604888 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.604865    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-openshift-service-ca\") pod \"maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc\" (UID: \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"
Oct 30 20:03:47.606625 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.606599    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-maas-api-tls\") pod \"maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc\" (UID: \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"
Oct 30 20:03:47.606829 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.606810    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-metrics-certs\") pod \"maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc\" (UID: \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"
Oct 30 20:03:47.621124 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.621069    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwvx7\" (UniqueName: \"kubernetes.io/projected/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-kube-api-access-nwvx7\") pod \"maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc\" (UID: \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"
Oct 30 20:03:47.685905 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:47.685879    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"
Oct 30 20:03:48.159053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:48.158357    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"]
Oct 30 20:03:48.235052 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:03:48.231078    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc2d2e5_3aab_4f01_997b_07aa7841dd73.slice/crio-ee96bd98ad0a3ac5fe09e93861ef8a80460662b7d87dedbd0522552b7a786ae0 WatchSource:0}: Error finding container ee96bd98ad0a3ac5fe09e93861ef8a80460662b7d87dedbd0522552b7a786ae0: Status 404 returned error can't find the container with id ee96bd98ad0a3ac5fe09e93861ef8a80460662b7d87dedbd0522552b7a786ae0
Oct 30 20:03:49.013559 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:49.013514    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb"]
Oct 30 20:03:49.020348 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:49.020320    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb"
Oct 30 20:03:49.260148 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:49.139806    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-466qk\" (UniqueName: \"kubernetes.io/projected/f877a279-f254-4591-a1ee-fa1804d8566b-kube-api-access-466qk\") pod \"payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb\" (UID: \"f877a279-f254-4591-a1ee-fa1804d8566b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb"
Oct 30 20:03:49.260148 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:49.140001    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f877a279-f254-4591-a1ee-fa1804d8566b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb\" (UID: \"f877a279-f254-4591-a1ee-fa1804d8566b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb"
Oct 30 20:03:49.260148 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:49.188480    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1f5eee-dockercfg-76ns8\""
Oct 30 20:03:49.260148 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:49.193564    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1f5eee\""
Oct 30 20:03:49.260148 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:49.242323    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-466qk\" (UniqueName: \"kubernetes.io/projected/f877a279-f254-4591-a1ee-fa1804d8566b-kube-api-access-466qk\") pod \"payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb\" (UID: \"f877a279-f254-4591-a1ee-fa1804d8566b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb"
Oct 30 20:03:49.260148 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:49.242503    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f877a279-f254-4591-a1ee-fa1804d8566b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb\" (UID: \"f877a279-f254-4591-a1ee-fa1804d8566b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb"
Oct 30 20:03:49.260148 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:49.243694    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f877a279-f254-4591-a1ee-fa1804d8566b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb\" (UID: \"f877a279-f254-4591-a1ee-fa1804d8566b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb"
Oct 30 20:03:49.347668 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:49.346810    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb"]
Oct 30 20:03:49.387112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:49.362011    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-466qk\" (UniqueName: \"kubernetes.io/projected/f877a279-f254-4591-a1ee-fa1804d8566b-kube-api-access-466qk\") pod \"payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb\" (UID: \"f877a279-f254-4591-a1ee-fa1804d8566b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb"
Oct 30 20:03:49.443283 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:49.443176    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb"
Oct 30 20:03:49.445436 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:49.445405    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc" event={"ID":"3fc2d2e5-3aab-4f01-997b-07aa7841dd73","Type":"ContainerStarted","Data":"d46cc2f75c49f61cf5c9a62bcb8c58b3f142fb3089388e48d2334789f92db9d0"}
Oct 30 20:03:49.459533 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:49.459430    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"
Oct 30 20:03:49.459533 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:49.459475    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc" event={"ID":"3fc2d2e5-3aab-4f01-997b-07aa7841dd73","Type":"ContainerStarted","Data":"ee96bd98ad0a3ac5fe09e93861ef8a80460662b7d87dedbd0522552b7a786ae0"}
Oct 30 20:03:49.519605 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:49.513579    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc" podStartSLOduration=1.90516067 podStartE2EDuration="2.513559537s" totalImagesPullingTime="608.398867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:49.513088266 +0000 UTC m=+1984.097357884" watchObservedRunningTime="2026-10-30 20:03:49.513559537 +0000 UTC m=+1984.097829155"
Oct 30 20:03:49.806207 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:03:49.806167    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf877a279_f254_4591_a1ee_fa1804d8566b.slice/crio-693001d51c78fd113a51235d488fb176ea4713d9ad39d80af4c42fadc3d65190 WatchSource:0}: Error finding container 693001d51c78fd113a51235d488fb176ea4713d9ad39d80af4c42fadc3d65190: Status 404 returned error can't find the container with id 693001d51c78fd113a51235d488fb176ea4713d9ad39d80af4c42fadc3d65190
Oct 30 20:03:49.806407 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:49.806383    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb"]
Oct 30 20:03:50.453844 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:50.453803    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb" event={"ID":"f877a279-f254-4591-a1ee-fa1804d8566b","Type":"ContainerStarted","Data":"0d8f67ef01e9803c08bae4de130203968cbac10435d4d9e34a42bcedc4ed4762"}
Oct 30 20:03:50.454271 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:50.454071    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb"
Oct 30 20:03:50.454271 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:50.454115    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb" event={"ID":"f877a279-f254-4591-a1ee-fa1804d8566b","Type":"ContainerStarted","Data":"693001d51c78fd113a51235d488fb176ea4713d9ad39d80af4c42fadc3d65190"}
Oct 30 20:03:50.503881 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:50.503834    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb" podStartSLOduration=2.503820178 podStartE2EDuration="2.503820178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:50.499999325 +0000 UTC m=+1985.084268941" watchObservedRunningTime="2026-10-30 20:03:50.503820178 +0000 UTC m=+1985.088089811"
Oct 30 20:03:53.583567 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:53.583531    5408 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","podd8634c36-96d5-4ce4-b82f-fd4fd7a6b481"] err="unable to destroy cgroup paths for cgroup [kubepods burstable podd8634c36-96d5-4ce4-b82f-fd4fd7a6b481] : Timed out while waiting for systemd to remove kubepods-burstable-podd8634c36_96d5_4ce4_b82f_fd4fd7a6b481.slice"
Oct 30 20:03:53.583567 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:53.583562    5408 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="d8634c36-96d5-4ce4-b82f-fd4fd7a6b481" err="failed to delete cgroup paths for [kubepods burstable podd8634c36-96d5-4ce4-b82f-fd4fd7a6b481] : unable to destroy cgroup paths for cgroup [kubepods burstable podd8634c36-96d5-4ce4-b82f-fd4fd7a6b481] : Timed out while waiting for systemd to remove kubepods-burstable-podd8634c36_96d5_4ce4_b82f_fd4fd7a6b481.slice"
Oct 30 20:03:55.090235 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:55.090208    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7"
Oct 30 20:03:55.459412 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:55.459342    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"
Oct 30 20:03:56.145058 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:56.142551    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l"
Oct 30 20:03:57.753339 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.753236    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"]
Oct 30 20:03:57.765559 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.765465    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.783394 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.783346    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"]
Oct 30 20:03:57.784213 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.784182    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7f7d21-openshift-default-dockercfg-72nd2\""
Oct 30 20:03:57.827055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.826992    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-podinfo\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.827208 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.827189    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z92sv\" (UniqueName: \"kubernetes.io/projected/7c77d77b-fee8-48c4-a717-5c17f9f02453-kube-api-access-z92sv\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.827263 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.827229    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-workload-socket\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.827299 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.827262    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-envoy\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.827339 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.827325    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c77d77b-fee8-48c4-a717-5c17f9f02453-istiod-ca-cert\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.827382 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.827344    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-data\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.827382 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.827363    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-token\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.827465 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.827447    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-credential-socket\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.827506 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.827486    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-workload-certs\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.944053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.930128    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z92sv\" (UniqueName: \"kubernetes.io/projected/7c77d77b-fee8-48c4-a717-5c17f9f02453-kube-api-access-z92sv\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.944412 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.944378    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-workload-socket\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.944599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.944579    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-envoy\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.944819 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.944797    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c77d77b-fee8-48c4-a717-5c17f9f02453-istiod-ca-cert\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.944996 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.944957    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-data\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.945179 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.945162    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-token\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.945405 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.945388    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-credential-socket\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.945540 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.945527    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-workload-certs\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.945762 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.945746    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-podinfo\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:57.987084 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.946671    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c77d77b-fee8-48c4-a717-5c17f9f02453-istiod-ca-cert\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:58.009944 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.947264    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-workload-socket\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:58.010520 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.964692    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-data\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:58.010685 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.964929    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-credential-socket\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:58.010800 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:57.965214    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-workload-certs\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:58.011727 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:58.011696    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z92sv\" (UniqueName: \"kubernetes.io/projected/7c77d77b-fee8-48c4-a717-5c17f9f02453-kube-api-access-z92sv\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:58.012130 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:58.012109    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-envoy\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:58.024693 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:58.024638    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-token\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:58.033050 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:58.032998    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-podinfo\") pod \"e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") " pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:58.108806 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:58.108754    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:58.424501 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:58.424474    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"]
Oct 30 20:03:58.426165 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:03:58.426139    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c77d77b_fee8_48c4_a717_5c17f9f02453.slice/crio-3be1aa178968b57f84fed8d757ceba9ca54b840e0581fc0fdbffe14a2f6eb946 WatchSource:0}: Error finding container 3be1aa178968b57f84fed8d757ceba9ca54b840e0581fc0fdbffe14a2f6eb946: Status 404 returned error can't find the container with id 3be1aa178968b57f84fed8d757ceba9ca54b840e0581fc0fdbffe14a2f6eb946
Oct 30 20:03:58.429564 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:58.429519    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:03:58.429651 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:58.429610    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:03:58.429697 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:58.429653    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:03:58.808599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:58.808560    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2" event={"ID":"7c77d77b-fee8-48c4-a717-5c17f9f02453","Type":"ContainerStarted","Data":"a74979304917224f6fcabbd5f59699827335bed510c65d93c50424a5699b38f6"}
Oct 30 20:03:58.808999 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:58.808630    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2" event={"ID":"7c77d77b-fee8-48c4-a717-5c17f9f02453","Type":"ContainerStarted","Data":"3be1aa178968b57f84fed8d757ceba9ca54b840e0581fc0fdbffe14a2f6eb946"}
Oct 30 20:03:58.835921 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:58.835608    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2" podStartSLOduration=1.83559093 podStartE2EDuration="1.83559093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:58.835473877 +0000 UTC m=+1993.419743495" watchObservedRunningTime="2026-10-30 20:03:58.83559093 +0000 UTC m=+1993.419860549"
Oct 30 20:03:59.116511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:59.116119    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:03:59.123397 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:59.123371    5408 patch_prober.go:29] interesting pod/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 30 20:03:59.123579 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:03:59.123561    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2" podUID="7c77d77b-fee8-48c4-a717-5c17f9f02453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 30 20:04:00.116771 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:00.116738    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:04:00.830095 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:00.830061    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:04:00.831348 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:00.831325    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:04:01.466479 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.466443    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"]
Oct 30 20:04:01.478315 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.478287    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.482488 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.482467    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb"
Oct 30 20:04:01.488444 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.488417    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"]
Oct 30 20:04:01.491888 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.491866    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1f246b\"/\"kube-root-ca.crt\""
Oct 30 20:04:01.492699 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.492021    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1f246b\"/\"openshift-service-ca.crt\""
Oct 30 20:04:01.492699 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.492169    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1f246b\"/\"test-model-1f246b-kserve-self-signed-certs\""
Oct 30 20:04:01.492699 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.492260    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1f246b\"/\"default-dockercfg-fztp5\""
Oct 30 20:04:01.591633 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.591595    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-model-cache\") pod \"test-model-1f246b-kserve-7497b49b9c-j2vzh\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") " pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.591837 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.591658    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-home\") pod \"test-model-1f246b-kserve-7497b49b9c-j2vzh\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") " pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.591837 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.591686    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933df3f9-ffba-40a6-b455-4ddf13776704-tls-certs\") pod \"test-model-1f246b-kserve-7497b49b9c-j2vzh\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") " pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.591837 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.591823    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-tmp-dir\") pod \"test-model-1f246b-kserve-7497b49b9c-j2vzh\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") " pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.592013 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.591848    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97f87\" (UniqueName: \"kubernetes.io/projected/933df3f9-ffba-40a6-b455-4ddf13776704-kube-api-access-97f87\") pod \"test-model-1f246b-kserve-7497b49b9c-j2vzh\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") " pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.592970 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.592155    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-dshm\") pod \"test-model-1f246b-kserve-7497b49b9c-j2vzh\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") " pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.694090 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.694052    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-dshm\") pod \"test-model-1f246b-kserve-7497b49b9c-j2vzh\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") " pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.694427 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.694403    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-model-cache\") pod \"test-model-1f246b-kserve-7497b49b9c-j2vzh\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") " pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.694874 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.694853    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-home\") pod \"test-model-1f246b-kserve-7497b49b9c-j2vzh\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") " pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.695007 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.694988    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933df3f9-ffba-40a6-b455-4ddf13776704-tls-certs\") pod \"test-model-1f246b-kserve-7497b49b9c-j2vzh\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") " pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.695181 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.695115    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-tmp-dir\") pod \"test-model-1f246b-kserve-7497b49b9c-j2vzh\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") " pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.695181 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.695143    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97f87\" (UniqueName: \"kubernetes.io/projected/933df3f9-ffba-40a6-b455-4ddf13776704-kube-api-access-97f87\") pod \"test-model-1f246b-kserve-7497b49b9c-j2vzh\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") " pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.695323 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.694796    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-model-cache\") pod \"test-model-1f246b-kserve-7497b49b9c-j2vzh\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") " pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.695637 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.695611    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-tmp-dir\") pod \"test-model-1f246b-kserve-7497b49b9c-j2vzh\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") " pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.695969 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.695938    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-home\") pod \"test-model-1f246b-kserve-7497b49b9c-j2vzh\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") " pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.697919 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.697894    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933df3f9-ffba-40a6-b455-4ddf13776704-tls-certs\") pod \"test-model-1f246b-kserve-7497b49b9c-j2vzh\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") " pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.698174 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.698154    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-dshm\") pod \"test-model-1f246b-kserve-7497b49b9c-j2vzh\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") " pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.707723 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.707700    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97f87\" (UniqueName: \"kubernetes.io/projected/933df3f9-ffba-40a6-b455-4ddf13776704-kube-api-access-97f87\") pod \"test-model-1f246b-kserve-7497b49b9c-j2vzh\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") " pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:01.817053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:01.816799    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:02.417770 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:02.417746    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"]
Oct 30 20:04:02.419167 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:04:02.419143    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod933df3f9_ffba_40a6_b455_4ddf13776704.slice/crio-c9c02aa53084109039019a847991a78b633fb47037ac6aa55eff0ea469ad1962 WatchSource:0}: Error finding container c9c02aa53084109039019a847991a78b633fb47037ac6aa55eff0ea469ad1962: Status 404 returned error can't find the container with id c9c02aa53084109039019a847991a78b633fb47037ac6aa55eff0ea469ad1962
Oct 30 20:04:02.851602 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:02.851571    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg"]
Oct 30 20:04:02.856231 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:02.856203    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh" event={"ID":"933df3f9-ffba-40a6-b455-4ddf13776704","Type":"ContainerStarted","Data":"0923206460aa67d84e454170a730517cda7e113a6455c35905e084c8e8705e92"}
Oct 30 20:04:02.856351 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:02.856235    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh" event={"ID":"933df3f9-ffba-40a6-b455-4ddf13776704","Type":"ContainerStarted","Data":"c9c02aa53084109039019a847991a78b633fb47037ac6aa55eff0ea469ad1962"}
Oct 30 20:04:02.856351 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:02.856280    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg"
Oct 30 20:04:02.868422 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:02.866994    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7f7d21\""
Oct 30 20:04:02.868422 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:02.867066    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7f7d21-dockercfg-9v9wb\""
Oct 30 20:04:02.868422 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:02.867176    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg"]
Oct 30 20:04:02.906138 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:02.906084    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh" podStartSLOduration=1.9060715190000002 podStartE2EDuration="1.906071519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:02.90182256 +0000 UTC m=+1997.486092177" watchObservedRunningTime="2026-10-30 20:04:02.906071519 +0000 UTC m=+1997.490341148"
Oct 30 20:04:02.947866 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:02.947834    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2982e319-9261-44c2-bc78-54bd30e83dea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg\" (UID: \"2982e319-9261-44c2-bc78-54bd30e83dea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg"
Oct 30 20:04:02.948109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:02.948088    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cfj4\" (UniqueName: \"kubernetes.io/projected/2982e319-9261-44c2-bc78-54bd30e83dea-kube-api-access-2cfj4\") pod \"payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg\" (UID: \"2982e319-9261-44c2-bc78-54bd30e83dea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg"
Oct 30 20:04:03.061254 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:03.061199    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cfj4\" (UniqueName: \"kubernetes.io/projected/2982e319-9261-44c2-bc78-54bd30e83dea-kube-api-access-2cfj4\") pod \"payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg\" (UID: \"2982e319-9261-44c2-bc78-54bd30e83dea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg"
Oct 30 20:04:03.061726 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:03.061703    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2982e319-9261-44c2-bc78-54bd30e83dea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg\" (UID: \"2982e319-9261-44c2-bc78-54bd30e83dea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg"
Oct 30 20:04:03.082003 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:03.081967    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2982e319-9261-44c2-bc78-54bd30e83dea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg\" (UID: \"2982e319-9261-44c2-bc78-54bd30e83dea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg"
Oct 30 20:04:03.127090 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:03.126998    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cfj4\" (UniqueName: \"kubernetes.io/projected/2982e319-9261-44c2-bc78-54bd30e83dea-kube-api-access-2cfj4\") pod \"payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg\" (UID: \"2982e319-9261-44c2-bc78-54bd30e83dea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg"
Oct 30 20:04:03.194256 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:03.191610    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg"
Oct 30 20:04:03.839066 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:03.838953    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg"]
Oct 30 20:04:03.844051 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:03.841744    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:03.973048 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:04:03.970570    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2982e319_9261_44c2_bc78_54bd30e83dea.slice/crio-ea53f7acf22e67c726ab5e5601aab70cd222e196052fc1ce7e5c749d33ca7ac1 WatchSource:0}: Error finding container ea53f7acf22e67c726ab5e5601aab70cd222e196052fc1ce7e5c749d33ca7ac1: Status 404 returned error can't find the container with id ea53f7acf22e67c726ab5e5601aab70cd222e196052fc1ce7e5c749d33ca7ac1
Oct 30 20:04:04.942050 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:04.940556    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg"
Oct 30 20:04:04.942050 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:04.941667    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg" event={"ID":"2982e319-9261-44c2-bc78-54bd30e83dea","Type":"ContainerStarted","Data":"29bb6b68d12dab06f4b98d3a8ed4f1dd405a07d8f6afd96204bbf5bd314e89ec"}
Oct 30 20:04:04.942050 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:04.941714    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg" event={"ID":"2982e319-9261-44c2-bc78-54bd30e83dea","Type":"ContainerStarted","Data":"ea53f7acf22e67c726ab5e5601aab70cd222e196052fc1ce7e5c749d33ca7ac1"}
Oct 30 20:04:05.011050 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:05.010007    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg" podStartSLOduration=3.009985946 podStartE2EDuration="3.009985946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:05.005614343 +0000 UTC m=+1999.589883961" watchObservedRunningTime="2026-10-30 20:04:05.009985946 +0000 UTC m=+1999.594255565"
Oct 30 20:04:06.274055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:06.273500    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-mkl2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:04:06.274055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:06.273570    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p" podUID="6bf03122-99f0-4878-98ef-eaa9325402f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:04:06.635113 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:06.634104    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:04:06.676054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:06.675114    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:04:07.062056 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:07.061163    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:04:07.496108 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:07.493612    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-2wd98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:04:07.496108 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:07.493679    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98" podUID="adfef3ed-77c2-4817-9a2c-96403e22f83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:04:09.674632 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:04:09.674567    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:04:11.817683 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:11.817651    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:11.830354 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:11.830328    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:12.138018 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:12.137927    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:04:16.500480 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:16.499078    5408 patch_prober.go:29] interesting pod/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": read tcp 10.134.0.2:46146->10.134.0.140:15021: read: connection reset by peer" start-of-body=
Oct 30 20:04:16.500480 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:16.499154    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg" podUID="f99c11d2-540e-434a-8271-92ea41a6b437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": read tcp 10.134.0.2:46146->10.134.0.140:15021: read: connection reset by peer"
Oct 30 20:04:17.022336 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.022306    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg"
Oct 30 20:04:17.241103 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.241071    5408 generic.go:356] "Generic (PLEG): container finished" podID="f99c11d2-540e-434a-8271-92ea41a6b437" containerID="f0ec32cfcaf47fd6034c05d17dbafedead82651b9fa951537c0397927bab4d32" exitCode=0
Oct 30 20:04:17.241294 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.241144    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg" event={"ID":"f99c11d2-540e-434a-8271-92ea41a6b437","Type":"ContainerDied","Data":"f0ec32cfcaf47fd6034c05d17dbafedead82651b9fa951537c0397927bab4d32"}
Oct 30 20:04:17.241615 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.241601    5408 scope.go:122] "RemoveContainer" containerID="f0ec32cfcaf47fd6034c05d17dbafedead82651b9fa951537c0397927bab4d32"
Oct 30 20:04:17.243550 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.243518    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:04:17.243683 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.243601    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:04:17.243683 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.243650    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:04:17.457160 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.457127    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"]
Oct 30 20:04:17.461572 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.461537    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"
Oct 30 20:04:17.474448 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.474422    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1f5eee-serving-cert\""
Oct 30 20:04:17.503717 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.503131    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"]
Oct 30 20:04:17.562499 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.562414    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16a79c80-771e-418e-80ca-af817df5d495-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk\" (UID: \"16a79c80-771e-418e-80ca-af817df5d495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"
Oct 30 20:04:17.562668 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.562635    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a79c80-771e-418e-80ca-af817df5d495-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk\" (UID: \"16a79c80-771e-418e-80ca-af817df5d495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"
Oct 30 20:04:17.562748 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.562729    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9x7w\" (UniqueName: \"kubernetes.io/projected/16a79c80-771e-418e-80ca-af817df5d495-kube-api-access-k9x7w\") pod \"maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk\" (UID: \"16a79c80-771e-418e-80ca-af817df5d495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"
Oct 30 20:04:17.562852 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.562838    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16a79c80-771e-418e-80ca-af817df5d495-metrics-certs\") pod \"maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk\" (UID: \"16a79c80-771e-418e-80ca-af817df5d495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"
Oct 30 20:04:17.699783 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.697267    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16a79c80-771e-418e-80ca-af817df5d495-metrics-certs\") pod \"maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk\" (UID: \"16a79c80-771e-418e-80ca-af817df5d495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"
Oct 30 20:04:17.699783 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.697485    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16a79c80-771e-418e-80ca-af817df5d495-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk\" (UID: \"16a79c80-771e-418e-80ca-af817df5d495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"
Oct 30 20:04:17.699783 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.697816    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a79c80-771e-418e-80ca-af817df5d495-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk\" (UID: \"16a79c80-771e-418e-80ca-af817df5d495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"
Oct 30 20:04:17.699783 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.697893    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9x7w\" (UniqueName: \"kubernetes.io/projected/16a79c80-771e-418e-80ca-af817df5d495-kube-api-access-k9x7w\") pod \"maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk\" (UID: \"16a79c80-771e-418e-80ca-af817df5d495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"
Oct 30 20:04:17.709372 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.705002    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a79c80-771e-418e-80ca-af817df5d495-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk\" (UID: \"16a79c80-771e-418e-80ca-af817df5d495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"
Oct 30 20:04:17.753953 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.753860    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16a79c80-771e-418e-80ca-af817df5d495-metrics-certs\") pod \"maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk\" (UID: \"16a79c80-771e-418e-80ca-af817df5d495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"
Oct 30 20:04:17.773869 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.773798    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16a79c80-771e-418e-80ca-af817df5d495-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk\" (UID: \"16a79c80-771e-418e-80ca-af817df5d495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"
Oct 30 20:04:17.799306 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:17.799241    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9x7w\" (UniqueName: \"kubernetes.io/projected/16a79c80-771e-418e-80ca-af817df5d495-kube-api-access-k9x7w\") pod \"maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk\" (UID: \"16a79c80-771e-418e-80ca-af817df5d495\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"
Oct 30 20:04:18.093672 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:18.093634    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"
Oct 30 20:04:18.266052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:18.262328    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg" event={"ID":"f99c11d2-540e-434a-8271-92ea41a6b437","Type":"ContainerStarted","Data":"b911e63d830ac74f035c3c11fe0ef04eb5995e3bc7a44fa85247fb049e4a565e"}
Oct 30 20:04:18.739586 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:18.714539    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"]
Oct 30 20:04:18.844419 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:18.844381    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:04:18.844625 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:18.844596    5408 patch_prober.go:29] interesting pod/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:04:18.844709 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:18.844664    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg" podUID="f99c11d2-540e-434a-8271-92ea41a6b437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:04:19.271133 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:19.271097    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk" event={"ID":"16a79c80-771e-418e-80ca-af817df5d495","Type":"ContainerStarted","Data":"7dac98b4b7de9bb807f90ee02b2755371fed6f7f6c06e59283d197097b5089d0"}
Oct 30 20:04:19.849053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:19.846478    5408 patch_prober.go:29] interesting pod/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:04:19.849053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:19.846541    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg" podUID="f99c11d2-540e-434a-8271-92ea41a6b437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:04:20.281077 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:20.281019    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk" event={"ID":"16a79c80-771e-418e-80ca-af817df5d495","Type":"ContainerStarted","Data":"bc5a0962e9520d18afe3e8693e17c591b68fcd650dab209b0b1f9d7f0af9cfaf"}
Oct 30 20:04:20.281262 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:20.281127    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"
Oct 30 20:04:20.335112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:20.331763    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk" podStartSLOduration=2.763954591 podStartE2EDuration="3.331740245s" totalImagesPullingTime="567.785654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:20.305444651 +0000 UTC m=+2014.889714269" watchObservedRunningTime="2026-10-30 20:04:20.331740245 +0000 UTC m=+2014.916009863"
Oct 30 20:04:20.844401 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:20.844363    5408 patch_prober.go:29] interesting pod/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:04:20.844583 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:20.844417    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg" podUID="f99c11d2-540e-434a-8271-92ea41a6b437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:04:21.256403 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.256365    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"]
Oct 30 20:04:21.264679 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.264651    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.286093 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.273675    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-db02a7\"/\"default-dockercfg-rzfrb\""
Oct 30 20:04:21.286093 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.276243    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-db02a7\"/\"test-model-db02a7-kserve-self-signed-certs\""
Oct 30 20:04:21.288387 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.286716    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"]
Oct 30 20:04:21.327746 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.327712    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-db02a7\"/\"kube-root-ca.crt\""
Oct 30 20:04:21.348125 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.330278    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-db02a7\"/\"openshift-service-ca.crt\""
Oct 30 20:04:21.439062 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.436688    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-dshm\") pod \"test-model-db02a7-kserve-66fcd4c6bc-lf5ls\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") " pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.441094 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.439256    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-home\") pod \"test-model-db02a7-kserve-66fcd4c6bc-lf5ls\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") " pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.441094 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.439362    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5xm9\" (UniqueName: \"kubernetes.io/projected/f8b220f9-8403-4d34-a940-200f160de17f-kube-api-access-v5xm9\") pod \"test-model-db02a7-kserve-66fcd4c6bc-lf5ls\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") " pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.441094 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.439397    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-model-cache\") pod \"test-model-db02a7-kserve-66fcd4c6bc-lf5ls\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") " pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.441094 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.439420    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-tmp-dir\") pod \"test-model-db02a7-kserve-66fcd4c6bc-lf5ls\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") " pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.441094 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.439747    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b220f9-8403-4d34-a940-200f160de17f-tls-certs\") pod \"test-model-db02a7-kserve-66fcd4c6bc-lf5ls\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") " pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.546610 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.546518    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-dshm\") pod \"test-model-db02a7-kserve-66fcd4c6bc-lf5ls\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") " pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.546878 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.546847    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-home\") pod \"test-model-db02a7-kserve-66fcd4c6bc-lf5ls\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") " pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.547956 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.547461    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-home\") pod \"test-model-db02a7-kserve-66fcd4c6bc-lf5ls\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") " pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.548508 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.548166    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5xm9\" (UniqueName: \"kubernetes.io/projected/f8b220f9-8403-4d34-a940-200f160de17f-kube-api-access-v5xm9\") pod \"test-model-db02a7-kserve-66fcd4c6bc-lf5ls\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") " pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.559312 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.559286    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-model-cache\") pod \"test-model-db02a7-kserve-66fcd4c6bc-lf5ls\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") " pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.559513 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.559495    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-tmp-dir\") pod \"test-model-db02a7-kserve-66fcd4c6bc-lf5ls\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") " pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.559778 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.559763    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b220f9-8403-4d34-a940-200f160de17f-tls-certs\") pod \"test-model-db02a7-kserve-66fcd4c6bc-lf5ls\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") " pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.566070 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.562344    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-model-cache\") pod \"test-model-db02a7-kserve-66fcd4c6bc-lf5ls\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") " pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.566070 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.558559    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-dshm\") pod \"test-model-db02a7-kserve-66fcd4c6bc-lf5ls\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") " pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.566070 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.562599    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-tmp-dir\") pod \"test-model-db02a7-kserve-66fcd4c6bc-lf5ls\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") " pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.595087 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.591766    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5xm9\" (UniqueName: \"kubernetes.io/projected/f8b220f9-8403-4d34-a940-200f160de17f-kube-api-access-v5xm9\") pod \"test-model-db02a7-kserve-66fcd4c6bc-lf5ls\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") " pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.603174 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.602900    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b220f9-8403-4d34-a940-200f160de17f-tls-certs\") pod \"test-model-db02a7-kserve-66fcd4c6bc-lf5ls\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") " pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:21.844526 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.844392    5408 patch_prober.go:29] interesting pod/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:04:21.844526 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.844462    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg" podUID="f99c11d2-540e-434a-8271-92ea41a6b437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:04:21.898339 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:21.898290    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:22.260438 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:22.260407    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"]
Oct 30 20:04:22.262977 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:04:22.262942    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8b220f9_8403_4d34_a940_200f160de17f.slice/crio-df50d14ef0012ccf7476e13b17b9d2dde94678c9b8070d3517deba6145c69ace WatchSource:0}: Error finding container df50d14ef0012ccf7476e13b17b9d2dde94678c9b8070d3517deba6145c69ace: Status 404 returned error can't find the container with id df50d14ef0012ccf7476e13b17b9d2dde94678c9b8070d3517deba6145c69ace
Oct 30 20:04:22.346339 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:22.346296    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls" event={"ID":"f8b220f9-8403-4d34-a940-200f160de17f","Type":"ContainerStarted","Data":"df50d14ef0012ccf7476e13b17b9d2dde94678c9b8070d3517deba6145c69ace"}
Oct 30 20:04:22.844635 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:22.844587    5408 patch_prober.go:29] interesting pod/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:04:22.844849 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:22.844652    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg" podUID="f99c11d2-540e-434a-8271-92ea41a6b437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:04:23.424154 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:23.410400    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls" event={"ID":"f8b220f9-8403-4d34-a940-200f160de17f","Type":"ContainerStarted","Data":"b80f917f8e6390feae84b3beac0c737c7ea5732411d7dbfe4eeca22579200808"}
Oct 30 20:04:23.462193 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:23.462124    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls" podStartSLOduration=2.462099524 podStartE2EDuration="2.462099524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:23.438271225 +0000 UTC m=+2018.022540844" watchObservedRunningTime="2026-10-30 20:04:23.462099524 +0000 UTC m=+2018.046369143"
Oct 30 20:04:23.857264 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:23.856557    5408 patch_prober.go:29] interesting pod/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:04:23.857264 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:23.856611    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg" podUID="f99c11d2-540e-434a-8271-92ea41a6b437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:04:23.900584 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:23.900534    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:24.849163 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:24.849130    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:04:24.849634 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:24.849555    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:04:24.850548 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:24.850528    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:04:26.326683 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:26.326650    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"
Oct 30 20:04:26.877192 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:26.877156    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"]
Oct 30 20:04:26.882386 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:26.882359    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:26.891678 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:26.891651    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4f244d\"/\"test-model-4f244d-kserve-self-signed-certs\""
Oct 30 20:04:26.891872 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:26.891666    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4f244d\"/\"kube-root-ca.crt\""
Oct 30 20:04:26.891983 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:26.891705    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4f244d\"/\"openshift-service-ca.crt\""
Oct 30 20:04:26.891983 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:26.891767    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4f244d\"/\"default-dockercfg-8ptwh\""
Oct 30 20:04:26.892323 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:26.892301    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"]
Oct 30 20:04:26.935877 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:26.935848    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3cbd0cd-db30-4181-9271-fab50895d832-tls-certs\") pod \"test-model-4f244d-kserve-6d9d7db879-jwrd2\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") " pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:26.936117 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:26.935887    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-home\") pod \"test-model-4f244d-kserve-6d9d7db879-jwrd2\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") " pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:26.936213 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:26.936117    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-model-cache\") pod \"test-model-4f244d-kserve-6d9d7db879-jwrd2\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") " pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:26.936213 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:26.936166    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jsvl\" (UniqueName: \"kubernetes.io/projected/d3cbd0cd-db30-4181-9271-fab50895d832-kube-api-access-8jsvl\") pod \"test-model-4f244d-kserve-6d9d7db879-jwrd2\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") " pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:26.936397 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:26.936374    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-tmp-dir\") pod \"test-model-4f244d-kserve-6d9d7db879-jwrd2\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") " pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:26.936549 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:26.936530    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-dshm\") pod \"test-model-4f244d-kserve-6d9d7db879-jwrd2\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") " pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:27.051053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:27.038421    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-dshm\") pod \"test-model-4f244d-kserve-6d9d7db879-jwrd2\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") " pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:27.060054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:27.051851    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3cbd0cd-db30-4181-9271-fab50895d832-tls-certs\") pod \"test-model-4f244d-kserve-6d9d7db879-jwrd2\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") " pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:27.060356 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:27.060319    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-home\") pod \"test-model-4f244d-kserve-6d9d7db879-jwrd2\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") " pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:27.068180 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:27.061407    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-home\") pod \"test-model-4f244d-kserve-6d9d7db879-jwrd2\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") " pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:27.068550 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:27.068522    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-model-cache\") pod \"test-model-4f244d-kserve-6d9d7db879-jwrd2\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") " pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:27.079203 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:27.079166    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jsvl\" (UniqueName: \"kubernetes.io/projected/d3cbd0cd-db30-4181-9271-fab50895d832-kube-api-access-8jsvl\") pod \"test-model-4f244d-kserve-6d9d7db879-jwrd2\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") " pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:27.079646 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:27.079620    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-tmp-dir\") pod \"test-model-4f244d-kserve-6d9d7db879-jwrd2\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") " pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:27.092063 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:27.080393    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-tmp-dir\") pod \"test-model-4f244d-kserve-6d9d7db879-jwrd2\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") " pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:27.106055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:27.082842    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3cbd0cd-db30-4181-9271-fab50895d832-tls-certs\") pod \"test-model-4f244d-kserve-6d9d7db879-jwrd2\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") " pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:27.106055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:27.068995    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-model-cache\") pod \"test-model-4f244d-kserve-6d9d7db879-jwrd2\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") " pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:27.106055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:27.105386    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-dshm\") pod \"test-model-4f244d-kserve-6d9d7db879-jwrd2\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") " pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:27.134100 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:27.122834    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jsvl\" (UniqueName: \"kubernetes.io/projected/d3cbd0cd-db30-4181-9271-fab50895d832-kube-api-access-8jsvl\") pod \"test-model-4f244d-kserve-6d9d7db879-jwrd2\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") " pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:27.205198 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:27.199567    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:27.655240 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:27.655209    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"]
Oct 30 20:04:27.655791 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:27.655502    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq" podUID="8fe054e1-8ae6-4066-a04c-94a49bddb6a2" containerName="istio-proxy" containerID="cri-o://29b95f6a3170c5c292a2f34b2c8a15443a74880f67aae138e30347ceab11111d" gracePeriod=30
Oct 30 20:04:27.780633 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:27.780596    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"]
Oct 30 20:04:28.307568 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:28.307532    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"]
Oct 30 20:04:28.307782 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:28.307743    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp" podUID="0a0737c2-db49-4cb0-b815-0b2c9560119e" containerName="istio-proxy" containerID="cri-o://b8c9cb07dc298fa35bc33d55eb0525e6a73d9a90a63989e437547c83dba168fb" gracePeriod=30
Oct 30 20:04:28.665105 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:28.659474    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2" event={"ID":"d3cbd0cd-db30-4181-9271-fab50895d832","Type":"ContainerStarted","Data":"4ffc965ca80919565296fbcdfeaab88b1d99309271dcbac45f553d1e9f7ae4eb"}
Oct 30 20:04:28.665105 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:28.660158    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2" event={"ID":"d3cbd0cd-db30-4181-9271-fab50895d832","Type":"ContainerStarted","Data":"1beb1c2cfba162011946d5272d476298ee578034706131b2a2b801c6a62c9ede"}
Oct 30 20:04:28.688215 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:28.687434    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2" podStartSLOduration=2.687408356 podStartE2EDuration="2.687408356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:28.683886122 +0000 UTC m=+2023.268155739" watchObservedRunningTime="2026-10-30 20:04:28.687408356 +0000 UTC m=+2023.271677974"
Oct 30 20:04:29.200282 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:29.200253    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:31.909052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:31.907408    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:31.973404 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:31.973371    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:32.747120 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:32.747094    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:04:32.997971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:32.997946    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 20:04:33.050075 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.050048    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-workload-socket\") pod \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") "
Oct 30 20:04:33.050240 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.050167    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-token\" (UniqueName: \"kubernetes.io/projected/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-token\") pod \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") "
Oct 30 20:04:33.050240 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.050204    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-podinfo\") pod \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") "
Oct 30 20:04:33.050370 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.050306    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-workload-certs\") pod \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") "
Oct 30 20:04:33.050430 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.050381    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-kube-api-access-w4slp\" (UniqueName: \"kubernetes.io/projected/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-kube-api-access-w4slp\") pod \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") "
Oct 30 20:04:33.050541 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.050528    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-envoy\") pod \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") "
Oct 30 20:04:33.050596 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.050569    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-data\") pod \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") "
Oct 30 20:04:33.050596 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.050580    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-workload-socket" pod "8fe054e1-8ae6-4066-a04c-94a49bddb6a2" (UID: "8fe054e1-8ae6-4066-a04c-94a49bddb6a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:33.050733 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.050622    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-credential-socket\") pod \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") "
Oct 30 20:04:33.050733 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.050687    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istiod-ca-cert\") pod \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\" (UID: \"8fe054e1-8ae6-4066-a04c-94a49bddb6a2\") "
Oct 30 20:04:33.050904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.050881    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-workload-certs" pod "8fe054e1-8ae6-4066-a04c-94a49bddb6a2" (UID: "8fe054e1-8ae6-4066-a04c-94a49bddb6a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:33.051115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.051078    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-credential-socket" pod "8fe054e1-8ae6-4066-a04c-94a49bddb6a2" (UID: "8fe054e1-8ae6-4066-a04c-94a49bddb6a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:33.051213 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.051121    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-data" pod "8fe054e1-8ae6-4066-a04c-94a49bddb6a2" (UID: "8fe054e1-8ae6-4066-a04c-94a49bddb6a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:33.051430 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.051406    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istiod-ca-cert" pod "8fe054e1-8ae6-4066-a04c-94a49bddb6a2" (UID: "8fe054e1-8ae6-4066-a04c-94a49bddb6a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:33.051908 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.051862    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.052129 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.052063    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.052129 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.052091    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.052364 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.052105    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.052364 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.052340    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.055570 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.055494    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-token" pod "8fe054e1-8ae6-4066-a04c-94a49bddb6a2" (UID: "8fe054e1-8ae6-4066-a04c-94a49bddb6a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:33.068171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.055935    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-envoy" pod "8fe054e1-8ae6-4066-a04c-94a49bddb6a2" (UID: "8fe054e1-8ae6-4066-a04c-94a49bddb6a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:33.068171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.055987    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-kube-api-access-w4slp" pod "8fe054e1-8ae6-4066-a04c-94a49bddb6a2" (UID: "8fe054e1-8ae6-4066-a04c-94a49bddb6a2"). InnerVolumeSpecName "kube-api-access-w4slp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:33.068171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.056278    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-podinfo" pod "8fe054e1-8ae6-4066-a04c-94a49bddb6a2" (UID: "8fe054e1-8ae6-4066-a04c-94a49bddb6a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:04:33.166082 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.166002    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.166082 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.166056    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.166082 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.166072    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4slp\" (UniqueName: \"kubernetes.io/projected/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-kube-api-access-w4slp\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.166082 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.166089    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe054e1-8ae6-4066-a04c-94a49bddb6a2-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.596124 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.596098    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 20:04:33.706052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.696574    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-workload-socket\") pod \"0a0737c2-db49-4cb0-b815-0b2c9560119e\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") "
Oct 30 20:04:33.706052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.696711    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-data\") pod \"0a0737c2-db49-4cb0-b815-0b2c9560119e\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") "
Oct 30 20:04:33.706052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.696820    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-token\" (UniqueName: \"kubernetes.io/projected/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-token\") pod \"0a0737c2-db49-4cb0-b815-0b2c9560119e\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") "
Oct 30 20:04:33.706052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.696920    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-workload-certs\") pod \"0a0737c2-db49-4cb0-b815-0b2c9560119e\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") "
Oct 30 20:04:33.706052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.696967    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0737c2-db49-4cb0-b815-0b2c9560119e-kube-api-access-gp86j\" (UniqueName: \"kubernetes.io/projected/0a0737c2-db49-4cb0-b815-0b2c9560119e-kube-api-access-gp86j\") pod \"0a0737c2-db49-4cb0-b815-0b2c9560119e\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") "
Oct 30 20:04:33.706052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.696992    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a0737c2-db49-4cb0-b815-0b2c9560119e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0737c2-db49-4cb0-b815-0b2c9560119e-istiod-ca-cert\") pod \"0a0737c2-db49-4cb0-b815-0b2c9560119e\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") "
Oct 30 20:04:33.706052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.697081    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-credential-socket\") pod \"0a0737c2-db49-4cb0-b815-0b2c9560119e\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") "
Oct 30 20:04:33.706052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.697127    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-podinfo\") pod \"0a0737c2-db49-4cb0-b815-0b2c9560119e\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") "
Oct 30 20:04:33.706052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.697211    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-envoy\") pod \"0a0737c2-db49-4cb0-b815-0b2c9560119e\" (UID: \"0a0737c2-db49-4cb0-b815-0b2c9560119e\") "
Oct 30 20:04:33.706052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.697429    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-workload-certs" pod "0a0737c2-db49-4cb0-b815-0b2c9560119e" (UID: "0a0737c2-db49-4cb0-b815-0b2c9560119e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:33.706052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.698122    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.706052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.699280    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-data" pod "0a0737c2-db49-4cb0-b815-0b2c9560119e" (UID: "0a0737c2-db49-4cb0-b815-0b2c9560119e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:33.706052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.699679    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-credential-socket" pod "0a0737c2-db49-4cb0-b815-0b2c9560119e" (UID: "0a0737c2-db49-4cb0-b815-0b2c9560119e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:33.706052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.699849    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0737c2-db49-4cb0-b815-0b2c9560119e-istiod-ca-cert" pod "0a0737c2-db49-4cb0-b815-0b2c9560119e" (UID: "0a0737c2-db49-4cb0-b815-0b2c9560119e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:33.706052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.704103    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-workload-socket" pod "0a0737c2-db49-4cb0-b815-0b2c9560119e" (UID: "0a0737c2-db49-4cb0-b815-0b2c9560119e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:33.717748 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.717700    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-token" pod "0a0737c2-db49-4cb0-b815-0b2c9560119e" (UID: "0a0737c2-db49-4cb0-b815-0b2c9560119e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:33.718063 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.718006    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-podinfo" pod "0a0737c2-db49-4cb0-b815-0b2c9560119e" (UID: "0a0737c2-db49-4cb0-b815-0b2c9560119e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:04:33.719272 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.719243    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0737c2-db49-4cb0-b815-0b2c9560119e-kube-api-access-gp86j" pod "0a0737c2-db49-4cb0-b815-0b2c9560119e" (UID: "0a0737c2-db49-4cb0-b815-0b2c9560119e"). InnerVolumeSpecName "kube-api-access-gp86j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:33.719880 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.719849    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-envoy" pod "0a0737c2-db49-4cb0-b815-0b2c9560119e" (UID: "0a0737c2-db49-4cb0-b815-0b2c9560119e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:33.784604 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.783443    5408 generic.go:356] "Generic (PLEG): container finished" podID="8fe054e1-8ae6-4066-a04c-94a49bddb6a2" containerID="29b95f6a3170c5c292a2f34b2c8a15443a74880f67aae138e30347ceab11111d" exitCode=0
Oct 30 20:04:33.784604 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.783653    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq" event={"ID":"8fe054e1-8ae6-4066-a04c-94a49bddb6a2","Type":"ContainerDied","Data":"29b95f6a3170c5c292a2f34b2c8a15443a74880f67aae138e30347ceab11111d"}
Oct 30 20:04:33.784604 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.783705    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq" event={"ID":"8fe054e1-8ae6-4066-a04c-94a49bddb6a2","Type":"ContainerDied","Data":"4ae6f168557f60f9616dbb3887e89afa7d48a1e830ac38804750ee832eb90334"}
Oct 30 20:04:33.784604 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.783733    5408 scope.go:122] "RemoveContainer" containerID="29b95f6a3170c5c292a2f34b2c8a15443a74880f67aae138e30347ceab11111d"
Oct 30 20:04:33.784604 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.783902    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"
Oct 30 20:04:33.789761 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.789691    5408 generic.go:356] "Generic (PLEG): container finished" podID="0a0737c2-db49-4cb0-b815-0b2c9560119e" containerID="b8c9cb07dc298fa35bc33d55eb0525e6a73d9a90a63989e437547c83dba168fb" exitCode=0
Oct 30 20:04:33.804093 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.791263    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"
Oct 30 20:04:33.804093 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.795440    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp" event={"ID":"0a0737c2-db49-4cb0-b815-0b2c9560119e","Type":"ContainerDied","Data":"b8c9cb07dc298fa35bc33d55eb0525e6a73d9a90a63989e437547c83dba168fb"}
Oct 30 20:04:33.804093 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.795496    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp" event={"ID":"0a0737c2-db49-4cb0-b815-0b2c9560119e","Type":"ContainerDied","Data":"8c25a41cc53b8adafe3963c4762c908756910503158469ecbbea0e3937e7fcc1"}
Oct 30 20:04:33.804093 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.801344    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.804093 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.801372    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.804093 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.801386    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.804093 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.801400    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp86j\" (UniqueName: \"kubernetes.io/projected/0a0737c2-db49-4cb0-b815-0b2c9560119e-kube-api-access-gp86j\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.804093 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.801414    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0737c2-db49-4cb0-b815-0b2c9560119e-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.804093 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.801428    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.804093 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.801443    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.804093 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.801456    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0737c2-db49-4cb0-b815-0b2c9560119e-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.818072 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.818056    5408 scope.go:122] "RemoveContainer" containerID="29b95f6a3170c5c292a2f34b2c8a15443a74880f67aae138e30347ceab11111d"
Oct 30 20:04:33.818405 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:04:33.818368    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29b95f6a3170c5c292a2f34b2c8a15443a74880f67aae138e30347ceab11111d\": container with ID starting with 29b95f6a3170c5c292a2f34b2c8a15443a74880f67aae138e30347ceab11111d not found: ID does not exist" containerID="29b95f6a3170c5c292a2f34b2c8a15443a74880f67aae138e30347ceab11111d"
Oct 30 20:04:33.818459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.818427    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29b95f6a3170c5c292a2f34b2c8a15443a74880f67aae138e30347ceab11111d"} err="failed to get container status \"29b95f6a3170c5c292a2f34b2c8a15443a74880f67aae138e30347ceab11111d\": rpc error: code = NotFound desc = could not find container \"29b95f6a3170c5c292a2f34b2c8a15443a74880f67aae138e30347ceab11111d\": container with ID starting with 29b95f6a3170c5c292a2f34b2c8a15443a74880f67aae138e30347ceab11111d not found: ID does not exist"
Oct 30 20:04:33.818459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.818445    5408 scope.go:122] "RemoveContainer" containerID="b8c9cb07dc298fa35bc33d55eb0525e6a73d9a90a63989e437547c83dba168fb"
Oct 30 20:04:33.830377 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.830340    5408 scope.go:122] "RemoveContainer" containerID="b8c9cb07dc298fa35bc33d55eb0525e6a73d9a90a63989e437547c83dba168fb"
Oct 30 20:04:33.830665 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:04:33.830644    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c9cb07dc298fa35bc33d55eb0525e6a73d9a90a63989e437547c83dba168fb\": container with ID starting with b8c9cb07dc298fa35bc33d55eb0525e6a73d9a90a63989e437547c83dba168fb not found: ID does not exist" containerID="b8c9cb07dc298fa35bc33d55eb0525e6a73d9a90a63989e437547c83dba168fb"
Oct 30 20:04:33.830741 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.830675    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c9cb07dc298fa35bc33d55eb0525e6a73d9a90a63989e437547c83dba168fb"} err="failed to get container status \"b8c9cb07dc298fa35bc33d55eb0525e6a73d9a90a63989e437547c83dba168fb\": rpc error: code = NotFound desc = could not find container \"b8c9cb07dc298fa35bc33d55eb0525e6a73d9a90a63989e437547c83dba168fb\": container with ID starting with b8c9cb07dc298fa35bc33d55eb0525e6a73d9a90a63989e437547c83dba168fb not found: ID does not exist"
Oct 30 20:04:33.864118 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.864064    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"]
Oct 30 20:04:33.873120 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.873095    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-gndmp"]
Oct 30 20:04:33.914871 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.914843    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"]
Oct 30 20:04:33.934628 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:33.934602    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rllfq"]
Oct 30 20:04:34.114856 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.114781    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"]
Oct 30 20:04:34.115616 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.115599    5408 cpu_manager.go:414] "containerMap: removing container" podUID="0a0737c2-db49-4cb0-b815-0b2c9560119e" containerName="istio-proxy"
Oct 30 20:04:34.115672 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.115617    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0737c2-db49-4cb0-b815-0b2c9560119e" containerName="istio-proxy"
Oct 30 20:04:34.115672 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.115649    5408 cpu_manager.go:414] "containerMap: removing container" podUID="8fe054e1-8ae6-4066-a04c-94a49bddb6a2" containerName="istio-proxy"
Oct 30 20:04:34.115672 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.115653    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fe054e1-8ae6-4066-a04c-94a49bddb6a2" containerName="istio-proxy"
Oct 30 20:04:34.115778 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.115768    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fe054e1-8ae6-4066-a04c-94a49bddb6a2" containerName="istio-proxy"
Oct 30 20:04:34.115813 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.115787    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0737c2-db49-4cb0-b815-0b2c9560119e" containerName="istio-proxy"
Oct 30 20:04:34.121433 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.121410    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.143231 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.143194    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-77b4df\"/\"kube-root-ca.crt\""
Oct 30 20:04:34.143231 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.143216    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-77b4df\"/\"openshift-service-ca.crt\""
Oct 30 20:04:34.143231 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.143196    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-77b4df\"/\"test-model-77b4df-kserve-self-signed-certs\""
Oct 30 20:04:34.143439 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.143252    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-77b4df\"/\"default-dockercfg-pq5mz\""
Oct 30 20:04:34.161975 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.161948    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"]
Oct 30 20:04:34.207829 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.207797    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-tmp-dir\") pod \"test-model-77b4df-kserve-95796d7f-2t7dd\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") " pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.208025 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.208006    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-home\") pod \"test-model-77b4df-kserve-95796d7f-2t7dd\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") " pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.208144 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.208126    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-model-cache\") pod \"test-model-77b4df-kserve-95796d7f-2t7dd\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") " pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.208224 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.208210    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz998\" (UniqueName: \"kubernetes.io/projected/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-kube-api-access-jz998\") pod \"test-model-77b4df-kserve-95796d7f-2t7dd\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") " pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.208281 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.208264    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-tls-certs\") pod \"test-model-77b4df-kserve-95796d7f-2t7dd\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") " pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.208347 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.208332    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-dshm\") pod \"test-model-77b4df-kserve-95796d7f-2t7dd\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") " pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.360053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.313237    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-home\") pod \"test-model-77b4df-kserve-95796d7f-2t7dd\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") " pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.360053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.313408    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-model-cache\") pod \"test-model-77b4df-kserve-95796d7f-2t7dd\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") " pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.360053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.313488    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz998\" (UniqueName: \"kubernetes.io/projected/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-kube-api-access-jz998\") pod \"test-model-77b4df-kserve-95796d7f-2t7dd\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") " pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.360053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.313541    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-tls-certs\") pod \"test-model-77b4df-kserve-95796d7f-2t7dd\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") " pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.360053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.313606    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-dshm\") pod \"test-model-77b4df-kserve-95796d7f-2t7dd\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") " pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.360053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.313742    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-tmp-dir\") pod \"test-model-77b4df-kserve-95796d7f-2t7dd\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") " pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.360053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.314298    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-tmp-dir\") pod \"test-model-77b4df-kserve-95796d7f-2t7dd\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") " pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.360053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.314564    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-home\") pod \"test-model-77b4df-kserve-95796d7f-2t7dd\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") " pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.360053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.314801    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-model-cache\") pod \"test-model-77b4df-kserve-95796d7f-2t7dd\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") " pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.372893 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.372757    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-tls-certs\") pod \"test-model-77b4df-kserve-95796d7f-2t7dd\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") " pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.401109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.394987    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz998\" (UniqueName: \"kubernetes.io/projected/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-kube-api-access-jz998\") pod \"test-model-77b4df-kserve-95796d7f-2t7dd\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") " pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.405796 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.405729    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-dshm\") pod \"test-model-77b4df-kserve-95796d7f-2t7dd\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") " pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.452816 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.452775    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:34.824543 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:34.824508    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"]
Oct 30 20:04:34.824729 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:04:34.824706    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb19efa61_42ee_40ce_98b4_6b57dd46b3c1.slice/crio-bb1085608fdde35f3fff6f16484adbca54a64aa653c5be0c234dc459282274d3 WatchSource:0}: Error finding container bb1085608fdde35f3fff6f16484adbca54a64aa653c5be0c234dc459282274d3: Status 404 returned error can't find the container with id bb1085608fdde35f3fff6f16484adbca54a64aa653c5be0c234dc459282274d3
Oct 30 20:04:35.550695 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:35.550663    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a0737c2-db49-4cb0-b815-0b2c9560119e" path="/var/lib/kubelet/pods/0a0737c2-db49-4cb0-b815-0b2c9560119e/volumes"
Oct 30 20:04:35.552126 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:35.552097    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fe054e1-8ae6-4066-a04c-94a49bddb6a2" path="/var/lib/kubelet/pods/8fe054e1-8ae6-4066-a04c-94a49bddb6a2/volumes"
Oct 30 20:04:35.810490 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:35.810391    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd" event={"ID":"b19efa61-42ee-40ce-98b4-6b57dd46b3c1","Type":"ContainerStarted","Data":"4f0bb9358d5261433674e74212145cec12696d210839231ac54e6def43b495dc"}
Oct 30 20:04:35.810490 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:35.810435    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd" event={"ID":"b19efa61-42ee-40ce-98b4-6b57dd46b3c1","Type":"ContainerStarted","Data":"bb1085608fdde35f3fff6f16484adbca54a64aa653c5be0c234dc459282274d3"}
Oct 30 20:04:35.835686 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:35.835633    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd" podStartSLOduration=1.83562001 podStartE2EDuration="1.83562001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:35.834781576 +0000 UTC m=+2030.419051193" watchObservedRunningTime="2026-10-30 20:04:35.83562001 +0000 UTC m=+2030.419889626"
Oct 30 20:04:36.453638 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:36.453610    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:37.201000 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:37.200969    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:37.214543 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:37.214516    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:37.920937 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:37.920902    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:04:38.834261 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:38.834221    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"]
Oct 30 20:04:38.841717 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:38.841674    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:38.847224 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:38.847189    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3a4f02\"/\"kube-root-ca.crt\""
Oct 30 20:04:38.847335 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:38.847312    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3a4f02\"/\"openshift-service-ca.crt\""
Oct 30 20:04:38.850545 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:38.850524    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"]
Oct 30 20:04:38.852469 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:38.852450    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3a4f02\"/\"default-dockercfg-m25ch\""
Oct 30 20:04:38.852469 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:38.852453    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3a4f02\"/\"test-model-3a4f02-kserve-self-signed-certs\""
Oct 30 20:04:38.938688 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:38.938659    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-dshm\") pod \"test-model-3a4f02-kserve-7d45cc4755-nqd29\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") " pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:38.938852 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:38.938826    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-model-cache\") pod \"test-model-3a4f02-kserve-7d45cc4755-nqd29\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") " pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:38.938906 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:38.938875    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-home\") pod \"test-model-3a4f02-kserve-7d45cc4755-nqd29\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") " pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:38.939069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:38.939024    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz5mt\" (UniqueName: \"kubernetes.io/projected/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-kube-api-access-hz5mt\") pod \"test-model-3a4f02-kserve-7d45cc4755-nqd29\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") " pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:38.939191 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:38.939177    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-tls-certs\") pod \"test-model-3a4f02-kserve-7d45cc4755-nqd29\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") " pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:38.939318 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:38.939301    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-tmp-dir\") pod \"test-model-3a4f02-kserve-7d45cc4755-nqd29\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") " pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:39.053054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:39.047054    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-tls-certs\") pod \"test-model-3a4f02-kserve-7d45cc4755-nqd29\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") " pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:39.053054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:39.047196    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-tmp-dir\") pod \"test-model-3a4f02-kserve-7d45cc4755-nqd29\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") " pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:39.053054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:39.047298    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-dshm\") pod \"test-model-3a4f02-kserve-7d45cc4755-nqd29\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") " pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:39.053054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:39.047417    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-model-cache\") pod \"test-model-3a4f02-kserve-7d45cc4755-nqd29\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") " pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:39.053054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:39.047447    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-home\") pod \"test-model-3a4f02-kserve-7d45cc4755-nqd29\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") " pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:39.053054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:39.047546    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz5mt\" (UniqueName: \"kubernetes.io/projected/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-kube-api-access-hz5mt\") pod \"test-model-3a4f02-kserve-7d45cc4755-nqd29\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") " pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:39.054633 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:39.054605    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-home\") pod \"test-model-3a4f02-kserve-7d45cc4755-nqd29\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") " pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:39.055080 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:39.055055    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-model-cache\") pod \"test-model-3a4f02-kserve-7d45cc4755-nqd29\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") " pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:39.061748 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:39.055372    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-tmp-dir\") pod \"test-model-3a4f02-kserve-7d45cc4755-nqd29\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") " pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:39.088094 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:39.083755    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-tls-certs\") pod \"test-model-3a4f02-kserve-7d45cc4755-nqd29\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") " pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:39.088515 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:39.088485    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-dshm\") pod \"test-model-3a4f02-kserve-7d45cc4755-nqd29\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") " pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:39.104055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:39.091353    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz5mt\" (UniqueName: \"kubernetes.io/projected/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-kube-api-access-hz5mt\") pod \"test-model-3a4f02-kserve-7d45cc4755-nqd29\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") " pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:39.162058 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:39.161588    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:39.605874 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:39.605847    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"]
Oct 30 20:04:39.607102 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:04:39.607078    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3558d905_d0a3_4b3f_8c5c_dac3353f4b1c.slice/crio-cff2a512f0d4f1505066790ab08a317d94bbc4d4409f5fad332f52581287a6c0 WatchSource:0}: Error finding container cff2a512f0d4f1505066790ab08a317d94bbc4d4409f5fad332f52581287a6c0: Status 404 returned error can't find the container with id cff2a512f0d4f1505066790ab08a317d94bbc4d4409f5fad332f52581287a6c0
Oct 30 20:04:39.700126 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:04:39.689219    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:04:39.957837 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:39.944355    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29" event={"ID":"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c","Type":"ContainerStarted","Data":"cff2a512f0d4f1505066790ab08a317d94bbc4d4409f5fad332f52581287a6c0"}
Oct 30 20:04:40.956686 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:40.956644    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29" event={"ID":"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c","Type":"ContainerStarted","Data":"15a59837dbd6ce9f48f96f7adfbebd4222060471eaf4a231e4f9e144ada8ea80"}
Oct 30 20:04:41.033237 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.033189    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29" podStartSLOduration=3.033175056 podStartE2EDuration="3.033175056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:41.031383047 +0000 UTC m=+2035.615652663" watchObservedRunningTime="2026-10-30 20:04:41.033175056 +0000 UTC m=+2035.617444714"
Oct 30 20:04:41.047374 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.047349    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"]
Oct 30 20:04:41.052023 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.052006    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.092198 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.092168    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-82885f\"/\"openshift-service-ca.crt\""
Oct 30 20:04:41.099103 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.099085    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-82885f\"/\"default-dockercfg-579cp\""
Oct 30 20:04:41.104589 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.104567    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-82885f\"/\"test-model-82885f-kserve-self-signed-certs\""
Oct 30 20:04:41.109627 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.109603    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"]
Oct 30 20:04:41.275109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.233521    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-82885f\"/\"kube-root-ca.crt\""
Oct 30 20:04:41.275109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.235311    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:41.275109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.236546    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwtgd\" (UniqueName: \"kubernetes.io/projected/723bb78d-b254-4cea-bf5e-6e5609a8a699-kube-api-access-rwtgd\") pod \"test-model-82885f-kserve-f5cf59b77-f845z\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") " pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.275109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.236595    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-model-cache\") pod \"test-model-82885f-kserve-f5cf59b77-f845z\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") " pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.275109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.236716    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-home\") pod \"test-model-82885f-kserve-f5cf59b77-f845z\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") " pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.275109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.236742    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/723bb78d-b254-4cea-bf5e-6e5609a8a699-tls-certs\") pod \"test-model-82885f-kserve-f5cf59b77-f845z\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") " pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.275109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.236783    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-dshm\") pod \"test-model-82885f-kserve-f5cf59b77-f845z\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") " pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.275884 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.275851    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-tmp-dir\") pod \"test-model-82885f-kserve-f5cf59b77-f845z\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") " pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.444089 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.440290    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-tmp-dir\") pod \"test-model-82885f-kserve-f5cf59b77-f845z\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") " pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.444089 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.440518    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwtgd\" (UniqueName: \"kubernetes.io/projected/723bb78d-b254-4cea-bf5e-6e5609a8a699-kube-api-access-rwtgd\") pod \"test-model-82885f-kserve-f5cf59b77-f845z\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") " pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.444089 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.440556    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-model-cache\") pod \"test-model-82885f-kserve-f5cf59b77-f845z\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") " pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.444089 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.440911    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-home\") pod \"test-model-82885f-kserve-f5cf59b77-f845z\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") " pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.444089 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.440946    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/723bb78d-b254-4cea-bf5e-6e5609a8a699-tls-certs\") pod \"test-model-82885f-kserve-f5cf59b77-f845z\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") " pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.444089 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.440991    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-dshm\") pod \"test-model-82885f-kserve-f5cf59b77-f845z\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") " pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.452051 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.450481    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-model-cache\") pod \"test-model-82885f-kserve-f5cf59b77-f845z\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") " pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.452051 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.450747    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-tmp-dir\") pod \"test-model-82885f-kserve-f5cf59b77-f845z\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") " pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.457046 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.452759    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-home\") pod \"test-model-82885f-kserve-f5cf59b77-f845z\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") " pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.466465 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.459754    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/723bb78d-b254-4cea-bf5e-6e5609a8a699-tls-certs\") pod \"test-model-82885f-kserve-f5cf59b77-f845z\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") " pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.466672 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.466622    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwtgd\" (UniqueName: \"kubernetes.io/projected/723bb78d-b254-4cea-bf5e-6e5609a8a699-kube-api-access-rwtgd\") pod \"test-model-82885f-kserve-f5cf59b77-f845z\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") " pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.471085 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.470333    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-dshm\") pod \"test-model-82885f-kserve-f5cf59b77-f845z\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") " pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:41.692117 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:41.690283    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:42.041204 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:42.041158    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"]
Oct 30 20:04:42.986997 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:42.986950    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z" event={"ID":"723bb78d-b254-4cea-bf5e-6e5609a8a699","Type":"ContainerStarted","Data":"e0fc6e8036e3c34dd9bd0f119e2f9afbc207e34650d206caaf4b4b9cf5e32f2e"}
Oct 30 20:04:42.986997 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:42.986994    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z" event={"ID":"723bb78d-b254-4cea-bf5e-6e5609a8a699","Type":"ContainerStarted","Data":"fc0cedb4340b2717f284bd697862e138bd2c730d8e893b9a513fd4280c702ba7"}
Oct 30 20:04:43.033024 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:43.032967    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z" podStartSLOduration=3.032954209 podStartE2EDuration="3.032954209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:43.031848517 +0000 UTC m=+2037.616118145" watchObservedRunningTime="2026-10-30 20:04:43.032954209 +0000 UTC m=+2037.617223867"
Oct 30 20:04:43.691940 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:43.691904    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:44.454208 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:44.454175    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:44.467057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:44.467016    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:45.014967 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:45.014935    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:04:49.177053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:49.170698    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:49.488263 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:49.488234    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:50.173656 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:50.173626    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:04:51.692095 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:51.692061    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:51.705756 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:51.705728    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:52.386402 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:52.386369    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:04:58.071447 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.068590    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"]
Oct 30 20:04:58.079232 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.078855    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.083676 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.083654    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"]
Oct 30 20:04:58.100260 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.100236    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a7bb6c\"/\"openshift-service-ca.crt\""
Oct 30 20:04:58.100365 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.100315    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a7bb6c\"/\"kube-root-ca.crt\""
Oct 30 20:04:58.116295 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.116272    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a7bb6c\"/\"default-dockercfg-8br8j\""
Oct 30 20:04:58.145332 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.145300    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a7bb6c\"/\"test-model-a7bb6c-kserve-self-signed-certs\""
Oct 30 20:04:58.167251 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.167203    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-tmp-dir\") pod \"test-model-a7bb6c-kserve-7b6df65d9-5wcfb\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") " pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.167534 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.167498    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b56dd57-5a11-4834-a128-5aa9e2385157-tls-certs\") pod \"test-model-a7bb6c-kserve-7b6df65d9-5wcfb\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") " pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.167645 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.167577    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-home\") pod \"test-model-a7bb6c-kserve-7b6df65d9-5wcfb\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") " pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.167728 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.167714    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-model-cache\") pod \"test-model-a7bb6c-kserve-7b6df65d9-5wcfb\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") " pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.167910 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.167891    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npbmf\" (UniqueName: \"kubernetes.io/projected/9b56dd57-5a11-4834-a128-5aa9e2385157-kube-api-access-npbmf\") pod \"test-model-a7bb6c-kserve-7b6df65d9-5wcfb\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") " pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.168060 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.168042    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-dshm\") pod \"test-model-a7bb6c-kserve-7b6df65d9-5wcfb\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") " pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.269226 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.269189    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npbmf\" (UniqueName: \"kubernetes.io/projected/9b56dd57-5a11-4834-a128-5aa9e2385157-kube-api-access-npbmf\") pod \"test-model-a7bb6c-kserve-7b6df65d9-5wcfb\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") " pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.269424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.269242    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-dshm\") pod \"test-model-a7bb6c-kserve-7b6df65d9-5wcfb\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") " pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.269424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.269303    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-tmp-dir\") pod \"test-model-a7bb6c-kserve-7b6df65d9-5wcfb\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") " pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.269542 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.269495    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b56dd57-5a11-4834-a128-5aa9e2385157-tls-certs\") pod \"test-model-a7bb6c-kserve-7b6df65d9-5wcfb\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") " pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.269608 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.269557    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-home\") pod \"test-model-a7bb6c-kserve-7b6df65d9-5wcfb\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") " pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.269665 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.269643    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-model-cache\") pod \"test-model-a7bb6c-kserve-7b6df65d9-5wcfb\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") " pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.269836 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.269811    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-tmp-dir\") pod \"test-model-a7bb6c-kserve-7b6df65d9-5wcfb\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") " pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.269905 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.269876    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-home\") pod \"test-model-a7bb6c-kserve-7b6df65d9-5wcfb\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") " pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.270163 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.270140    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-model-cache\") pod \"test-model-a7bb6c-kserve-7b6df65d9-5wcfb\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") " pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.272676 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.271695    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-dshm\") pod \"test-model-a7bb6c-kserve-7b6df65d9-5wcfb\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") " pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.272676 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.272226    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b56dd57-5a11-4834-a128-5aa9e2385157-tls-certs\") pod \"test-model-a7bb6c-kserve-7b6df65d9-5wcfb\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") " pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.282319 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.282296    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npbmf\" (UniqueName: \"kubernetes.io/projected/9b56dd57-5a11-4834-a128-5aa9e2385157-kube-api-access-npbmf\") pod \"test-model-a7bb6c-kserve-7b6df65d9-5wcfb\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") " pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:58.403111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:58.394903    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:04:59.058861 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:04:59.058816    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b56dd57_5a11_4834_a128_5aa9e2385157.slice/crio-6ce0e33ec38118cb7ca7a2e1627d6db8a73fdbed094251e7be406b9c09ffca33 WatchSource:0}: Error finding container 6ce0e33ec38118cb7ca7a2e1627d6db8a73fdbed094251e7be406b9c09ffca33: Status 404 returned error can't find the container with id 6ce0e33ec38118cb7ca7a2e1627d6db8a73fdbed094251e7be406b9c09ffca33
Oct 30 20:04:59.075059 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:59.074995    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"]
Oct 30 20:04:59.519278 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:59.515435    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb" event={"ID":"9b56dd57-5a11-4834-a128-5aa9e2385157","Type":"ContainerStarted","Data":"9961129148dfea086f748fdca856825bc06c50d422b7bf2e7be3d35af151160f"}
Oct 30 20:04:59.519278 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:59.515492    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb" event={"ID":"9b56dd57-5a11-4834-a128-5aa9e2385157","Type":"ContainerStarted","Data":"6ce0e33ec38118cb7ca7a2e1627d6db8a73fdbed094251e7be406b9c09ffca33"}
Oct 30 20:04:59.551711 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:04:59.551651    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb" podStartSLOduration=1.551638117 podStartE2EDuration="1.551638117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:59.54838231 +0000 UTC m=+2054.132651927" watchObservedRunningTime="2026-10-30 20:04:59.551638117 +0000 UTC m=+2054.135907752"
Oct 30 20:05:00.396725 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.396687    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:05:00.549681 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.549596    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"]
Oct 30 20:05:00.555775 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.555751    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.585481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.585444    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"]
Oct 30 20:05:00.636058 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.634819    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d7a733\"/\"test-model-d7a733-kserve-self-signed-certs\""
Oct 30 20:05:00.636058 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.634981    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d7a733\"/\"default-dockercfg-6txnq\""
Oct 30 20:05:00.636631 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.636600    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-dshm\") pod \"test-model-d7a733-kserve-5d55d59dcc-rbfbt\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") " pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.636786 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.636717    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d486b1-ecce-40a9-8353-04127a5111df-tls-certs\") pod \"test-model-d7a733-kserve-5d55d59dcc-rbfbt\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") " pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.636922 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.636902    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-tmp-dir\") pod \"test-model-d7a733-kserve-5d55d59dcc-rbfbt\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") " pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.637134 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.637112    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-home\") pod \"test-model-d7a733-kserve-5d55d59dcc-rbfbt\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") " pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.638324 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.638127    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4lz7\" (UniqueName: \"kubernetes.io/projected/90d486b1-ecce-40a9-8353-04127a5111df-kube-api-access-x4lz7\") pod \"test-model-d7a733-kserve-5d55d59dcc-rbfbt\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") " pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.638324 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.638177    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-model-cache\") pod \"test-model-d7a733-kserve-5d55d59dcc-rbfbt\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") " pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.657737 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.657710    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d7a733\"/\"openshift-service-ca.crt\""
Oct 30 20:05:00.659219 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.659198    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d7a733\"/\"kube-root-ca.crt\""
Oct 30 20:05:00.739955 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.739918    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d486b1-ecce-40a9-8353-04127a5111df-tls-certs\") pod \"test-model-d7a733-kserve-5d55d59dcc-rbfbt\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") " pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.740171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.740106    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-tmp-dir\") pod \"test-model-d7a733-kserve-5d55d59dcc-rbfbt\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") " pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.740260 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.740250    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-home\") pod \"test-model-d7a733-kserve-5d55d59dcc-rbfbt\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") " pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.740344 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.740324    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4lz7\" (UniqueName: \"kubernetes.io/projected/90d486b1-ecce-40a9-8353-04127a5111df-kube-api-access-x4lz7\") pod \"test-model-d7a733-kserve-5d55d59dcc-rbfbt\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") " pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.740417 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.740371    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-model-cache\") pod \"test-model-d7a733-kserve-5d55d59dcc-rbfbt\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") " pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.740475 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.740442    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-dshm\") pod \"test-model-d7a733-kserve-5d55d59dcc-rbfbt\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") " pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.740796 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.740761    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-home\") pod \"test-model-d7a733-kserve-5d55d59dcc-rbfbt\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") " pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.740935 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.740853    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-model-cache\") pod \"test-model-d7a733-kserve-5d55d59dcc-rbfbt\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") " pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.740935 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.740887    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-tmp-dir\") pod \"test-model-d7a733-kserve-5d55d59dcc-rbfbt\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") " pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.743969 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.743942    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-dshm\") pod \"test-model-d7a733-kserve-5d55d59dcc-rbfbt\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") " pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.744703 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.744681    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d486b1-ecce-40a9-8353-04127a5111df-tls-certs\") pod \"test-model-d7a733-kserve-5d55d59dcc-rbfbt\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") " pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.771588 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.771560    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4lz7\" (UniqueName: \"kubernetes.io/projected/90d486b1-ecce-40a9-8353-04127a5111df-kube-api-access-x4lz7\") pod \"test-model-d7a733-kserve-5d55d59dcc-rbfbt\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") " pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:00.879100 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:00.878672    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:01.679462 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:01.679434    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"]
Oct 30 20:05:01.682122 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:05:01.682093    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90d486b1_ecce_40a9_8353_04127a5111df.slice/crio-56ac3c2f32403cfc08769e498bbb26a9b8c35c897a9d56b4e9779631c59e67a3 WatchSource:0}: Error finding container 56ac3c2f32403cfc08769e498bbb26a9b8c35c897a9d56b4e9779631c59e67a3: Status 404 returned error can't find the container with id 56ac3c2f32403cfc08769e498bbb26a9b8c35c897a9d56b4e9779631c59e67a3
Oct 30 20:05:02.554854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:02.554819    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt" event={"ID":"90d486b1-ecce-40a9-8353-04127a5111df","Type":"ContainerStarted","Data":"b6eb2f9893863385ef82ae3b2cba570751cd78d69928011c4309985abe200de4"}
Oct 30 20:05:02.555045 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:02.554871    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt" event={"ID":"90d486b1-ecce-40a9-8353-04127a5111df","Type":"ContainerStarted","Data":"56ac3c2f32403cfc08769e498bbb26a9b8c35c897a9d56b4e9779631c59e67a3"}
Oct 30 20:05:02.582120 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:02.582061    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt" podStartSLOduration=2.582039479 podStartE2EDuration="2.582039479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:02.578663003 +0000 UTC m=+2057.162932621" watchObservedRunningTime="2026-10-30 20:05:02.582039479 +0000 UTC m=+2057.166309149"
Oct 30 20:05:02.879579 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:02.879506    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:05.498889 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:05.498859    5408 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 30 20:05:05.499392 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:05.498908    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v" podUID="50041f27-99a2-47ba-944a-9693459b8c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 30 20:05:05.584251 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:05.584168    5408 generic.go:356] "Generic (PLEG): container finished" podID="50041f27-99a2-47ba-944a-9693459b8c80" containerID="538e6c2c3732c0219e5b2d83dae3cf81bb5381664cece705c695da5c7ae4970b" exitCode=0
Oct 30 20:05:05.584424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:05.584241    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v" event={"ID":"50041f27-99a2-47ba-944a-9693459b8c80","Type":"ContainerDied","Data":"538e6c2c3732c0219e5b2d83dae3cf81bb5381664cece705c695da5c7ae4970b"}
Oct 30 20:05:05.584758 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:05.584739    5408 scope.go:122] "RemoveContainer" containerID="538e6c2c3732c0219e5b2d83dae3cf81bb5381664cece705c695da5c7ae4970b"
Oct 30 20:05:05.586880 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:05.586850    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:05:05.586984 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:05.586919    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:05:05.586984 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:05.586951    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:05:06.598884 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:06.596792    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v" event={"ID":"50041f27-99a2-47ba-944a-9693459b8c80","Type":"ContainerStarted","Data":"b60401a9e062499a42d892fe9cda49b560eefad2aabd529d8266bdc21cb1d880"}
Oct 30 20:05:06.923103 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:06.909117    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:05:06.923103 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:06.914152    5408 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 30 20:05:06.948054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:06.939318    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v" podUID="50041f27-99a2-47ba-944a-9693459b8c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 30 20:05:07.876743 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:07.875653    5408 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 30 20:05:07.876743 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:07.875714    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v" podUID="50041f27-99a2-47ba-944a-9693459b8c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 30 20:05:08.396247 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:08.396204    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:05:08.462924 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:08.462894    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:05:08.691446 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:08.691349    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:05:08.872675 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:08.872634    5408 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 30 20:05:08.872878 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:08.872699    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v" podUID="50041f27-99a2-47ba-944a-9693459b8c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 30 20:05:09.891063 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:09.890652    5408 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 30 20:05:09.891063 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:09.890722    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v" podUID="50041f27-99a2-47ba-944a-9693459b8c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 30 20:05:09.979841 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:05:09.979804    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:05:10.867639 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:10.867598    5408 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 30 20:05:10.867834 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:10.867673    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v" podUID="50041f27-99a2-47ba-944a-9693459b8c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 30 20:05:10.879791 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:10.879755    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:10.892972 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:10.892940    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:11.184489 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:11.184395    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:05:11.867615 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:11.867295    5408 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 30 20:05:11.867615 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:11.867352    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v" podUID="50041f27-99a2-47ba-944a-9693459b8c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 30 20:05:12.872264 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:12.872229    5408 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 30 20:05:12.873352 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:12.873317    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v" podUID="50041f27-99a2-47ba-944a-9693459b8c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 30 20:05:13.875001 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:13.874612    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:05:13.875701 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:13.875679    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:05:13.877593 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:13.877568    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:05:16.873204 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:16.873160    5408 patch_prober.go:29] interesting pod/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:05:16.873749 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:16.873218    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2" podUID="7c77d77b-fee8-48c4-a717-5c17f9f02453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:05:18.091787 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:18.091754    5408 generic.go:356] "Generic (PLEG): container finished" podID="7c77d77b-fee8-48c4-a717-5c17f9f02453" containerID="a74979304917224f6fcabbd5f59699827335bed510c65d93c50424a5699b38f6" exitCode=0
Oct 30 20:05:18.092319 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:18.091825    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2" event={"ID":"7c77d77b-fee8-48c4-a717-5c17f9f02453","Type":"ContainerDied","Data":"a74979304917224f6fcabbd5f59699827335bed510c65d93c50424a5699b38f6"}
Oct 30 20:05:18.092395 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:18.092341    5408 scope.go:122] "RemoveContainer" containerID="a74979304917224f6fcabbd5f59699827335bed510c65d93c50424a5699b38f6"
Oct 30 20:05:18.094436 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:18.094406    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:05:18.094525 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:18.094490    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:05:18.094564 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:18.094527    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:05:19.192047 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:19.184134    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2" event={"ID":"7c77d77b-fee8-48c4-a717-5c17f9f02453","Type":"ContainerStarted","Data":"ec614a3f56d6fa54ddd1dbeb4d05fef36074af35e9f4a0078ba28cf86c47a9f6"}
Oct 30 20:05:20.124054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:20.123900    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:05:20.124054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:20.124050    5408 patch_prober.go:29] interesting pod/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 30 20:05:20.124435 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:20.124107    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2" podUID="7c77d77b-fee8-48c4-a717-5c17f9f02453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 30 20:05:21.125055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:21.124817    5408 patch_prober.go:29] interesting pod/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 30 20:05:21.125055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:21.124889    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2" podUID="7c77d77b-fee8-48c4-a717-5c17f9f02453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 30 20:05:22.125052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:22.124189    5408 patch_prober.go:29] interesting pod/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 30 20:05:22.125052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:22.124249    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2" podUID="7c77d77b-fee8-48c4-a717-5c17f9f02453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 30 20:05:23.127055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:23.126133    5408 patch_prober.go:29] interesting pod/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 30 20:05:23.127055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:23.126205    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2" podUID="7c77d77b-fee8-48c4-a717-5c17f9f02453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 30 20:05:24.127057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:24.124426    5408 patch_prober.go:29] interesting pod/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 30 20:05:24.127057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:24.124504    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2" podUID="7c77d77b-fee8-48c4-a717-5c17f9f02453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 30 20:05:25.120052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:25.119365    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:05:25.124049 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:25.121383    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:05:25.146052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:25.139306    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:05:39.887105 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:05:39.875661    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:05:56.062515 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.062476    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"]
Oct 30 20:05:56.069824 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.069795    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.079897 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.079868    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"]
Oct 30 20:05:56.082372 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.082343    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-925e04\"/\"kube-root-ca.crt\""
Oct 30 20:05:56.082508 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.082466    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-925e04\"/\"openshift-service-ca.crt\""
Oct 30 20:05:56.084067 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.084007    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-925e04\"/\"test-model-925e04-kserve-self-signed-certs\""
Oct 30 20:05:56.084240 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.084172    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-925e04\"/\"default-dockercfg-mmrjm\""
Oct 30 20:05:56.100550 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.100520    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqr72\" (UniqueName: \"kubernetes.io/projected/8300c88b-76da-4c18-b214-49d9ae483bd7-kube-api-access-hqr72\") pod \"test-model-925e04-kserve-7969d4dd9c-hjbwf\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") " pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.100677 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.100592    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-home\") pod \"test-model-925e04-kserve-7969d4dd9c-hjbwf\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") " pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.100749 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.100731    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-model-cache\") pod \"test-model-925e04-kserve-7969d4dd9c-hjbwf\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") " pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.100798 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.100784    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8300c88b-76da-4c18-b214-49d9ae483bd7-tls-certs\") pod \"test-model-925e04-kserve-7969d4dd9c-hjbwf\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") " pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.100946 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.100926    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-tmp-dir\") pod \"test-model-925e04-kserve-7969d4dd9c-hjbwf\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") " pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.101153 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.101135    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-dshm\") pod \"test-model-925e04-kserve-7969d4dd9c-hjbwf\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") " pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.202724 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.202681    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-tmp-dir\") pod \"test-model-925e04-kserve-7969d4dd9c-hjbwf\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") " pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.202911 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.202819    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-dshm\") pod \"test-model-925e04-kserve-7969d4dd9c-hjbwf\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") " pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.203009 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.202992    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqr72\" (UniqueName: \"kubernetes.io/projected/8300c88b-76da-4c18-b214-49d9ae483bd7-kube-api-access-hqr72\") pod \"test-model-925e04-kserve-7969d4dd9c-hjbwf\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") " pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.203127 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.203110    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-home\") pod \"test-model-925e04-kserve-7969d4dd9c-hjbwf\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") " pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.203229 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.203205    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-model-cache\") pod \"test-model-925e04-kserve-7969d4dd9c-hjbwf\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") " pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.203229 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.203216    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-tmp-dir\") pod \"test-model-925e04-kserve-7969d4dd9c-hjbwf\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") " pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.203370 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.203282    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8300c88b-76da-4c18-b214-49d9ae483bd7-tls-certs\") pod \"test-model-925e04-kserve-7969d4dd9c-hjbwf\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") " pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.203528 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.203505    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-home\") pod \"test-model-925e04-kserve-7969d4dd9c-hjbwf\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") " pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.203613 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.203540    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-model-cache\") pod \"test-model-925e04-kserve-7969d4dd9c-hjbwf\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") " pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.205247 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.205215    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-dshm\") pod \"test-model-925e04-kserve-7969d4dd9c-hjbwf\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") " pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.205878 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.205856    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8300c88b-76da-4c18-b214-49d9ae483bd7-tls-certs\") pod \"test-model-925e04-kserve-7969d4dd9c-hjbwf\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") " pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.217555 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.217526    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqr72\" (UniqueName: \"kubernetes.io/projected/8300c88b-76da-4c18-b214-49d9ae483bd7-kube-api-access-hqr72\") pod \"test-model-925e04-kserve-7969d4dd9c-hjbwf\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") " pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.399107 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.398781    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:05:56.921666 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:56.921624    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"]
Oct 30 20:05:56.922693 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:05:56.922663    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8300c88b_76da_4c18_b214_49d9ae483bd7.slice/crio-a2774fc3b1833930d5bacb154ae336f03a91c51e05b016f1fed6777692d1146d WatchSource:0}: Error finding container a2774fc3b1833930d5bacb154ae336f03a91c51e05b016f1fed6777692d1146d: Status 404 returned error can't find the container with id a2774fc3b1833930d5bacb154ae336f03a91c51e05b016f1fed6777692d1146d
Oct 30 20:05:57.008653 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:57.008625    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf" event={"ID":"8300c88b-76da-4c18-b214-49d9ae483bd7","Type":"ContainerStarted","Data":"a2774fc3b1833930d5bacb154ae336f03a91c51e05b016f1fed6777692d1146d"}
Oct 30 20:05:58.019116 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:58.019075    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf" event={"ID":"8300c88b-76da-4c18-b214-49d9ae483bd7","Type":"ContainerStarted","Data":"371d82f04ce43bb10c457a6a1c3354d459d60d3bc4f9fd8ea45db9a92195d917"}
Oct 30 20:05:58.065053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:58.063404    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf" podStartSLOduration=2.063385948 podStartE2EDuration="2.063385948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:58.052558426 +0000 UTC m=+2112.636828044" watchObservedRunningTime="2026-10-30 20:05:58.063385948 +0000 UTC m=+2112.647655569"
Oct 30 20:05:58.400212 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:05:58.400126    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:06:06.399957 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:06.399924    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:06:06.412847 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:06.412821    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:06:07.106414 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:07.106383    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:06:09.705083 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:06:09.703745    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:06:14.386395 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.386349    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"]
Oct 30 20:06:14.391485 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.391457    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.398624 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.398575    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"]
Oct 30 20:06:14.405746 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.405703    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-19f2ee\"/\"default-dockercfg-p7fcf\""
Oct 30 20:06:14.406927 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.406900    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-19f2ee\"/\"test-model-19f2ee-kserve-self-signed-certs\""
Oct 30 20:06:14.486740 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.486706    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-19f2ee\"/\"openshift-service-ca.crt\""
Oct 30 20:06:14.486945 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.486706    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-19f2ee\"/\"kube-root-ca.crt\""
Oct 30 20:06:14.507687 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.507647    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-model-cache\") pod \"test-model-19f2ee-kserve-55549f997f-wwrmw\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") " pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.507874 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.507746    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-home\") pod \"test-model-19f2ee-kserve-55549f997f-wwrmw\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") " pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.507994 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.507968    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-dshm\") pod \"test-model-19f2ee-kserve-55549f997f-wwrmw\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") " pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.508157 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.508019    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154e4c02-a2b9-4dce-aec0-57c62527b2c4-tls-certs\") pod \"test-model-19f2ee-kserve-55549f997f-wwrmw\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") " pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.508314 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.508287    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f598s\" (UniqueName: \"kubernetes.io/projected/154e4c02-a2b9-4dce-aec0-57c62527b2c4-kube-api-access-f598s\") pod \"test-model-19f2ee-kserve-55549f997f-wwrmw\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") " pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.508384 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.508345    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-tmp-dir\") pod \"test-model-19f2ee-kserve-55549f997f-wwrmw\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") " pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.640114 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.635054    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-dshm\") pod \"test-model-19f2ee-kserve-55549f997f-wwrmw\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") " pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.640114 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.635147    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154e4c02-a2b9-4dce-aec0-57c62527b2c4-tls-certs\") pod \"test-model-19f2ee-kserve-55549f997f-wwrmw\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") " pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.640114 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.635459    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f598s\" (UniqueName: \"kubernetes.io/projected/154e4c02-a2b9-4dce-aec0-57c62527b2c4-kube-api-access-f598s\") pod \"test-model-19f2ee-kserve-55549f997f-wwrmw\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") " pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.640114 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.635537    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-tmp-dir\") pod \"test-model-19f2ee-kserve-55549f997f-wwrmw\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") " pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.640114 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.635635    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-model-cache\") pod \"test-model-19f2ee-kserve-55549f997f-wwrmw\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") " pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.640114 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.635707    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-home\") pod \"test-model-19f2ee-kserve-55549f997f-wwrmw\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") " pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.640114 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.636297    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-home\") pod \"test-model-19f2ee-kserve-55549f997f-wwrmw\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") " pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.644054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.640872    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-dshm\") pod \"test-model-19f2ee-kserve-55549f997f-wwrmw\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") " pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.644054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.641771    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-tmp-dir\") pod \"test-model-19f2ee-kserve-55549f997f-wwrmw\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") " pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.644054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.642188    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-model-cache\") pod \"test-model-19f2ee-kserve-55549f997f-wwrmw\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") " pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.674055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.670730    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f598s\" (UniqueName: \"kubernetes.io/projected/154e4c02-a2b9-4dce-aec0-57c62527b2c4-kube-api-access-f598s\") pod \"test-model-19f2ee-kserve-55549f997f-wwrmw\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") " pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.696053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.677712    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154e4c02-a2b9-4dce-aec0-57c62527b2c4-tls-certs\") pod \"test-model-19f2ee-kserve-55549f997f-wwrmw\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") " pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:14.718053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:14.714862    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:15.226325 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:06:15.226288    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod154e4c02_a2b9_4dce_aec0_57c62527b2c4.slice/crio-ddb7060e7e4cd50f66c392857d20efcddeea7da2a3901c43ad8b2ec75e809c2f WatchSource:0}: Error finding container ddb7060e7e4cd50f66c392857d20efcddeea7da2a3901c43ad8b2ec75e809c2f: Status 404 returned error can't find the container with id ddb7060e7e4cd50f66c392857d20efcddeea7da2a3901c43ad8b2ec75e809c2f
Oct 30 20:06:15.227276 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:15.227158    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"]
Oct 30 20:06:16.195643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.195570    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw" event={"ID":"154e4c02-a2b9-4dce-aec0-57c62527b2c4","Type":"ContainerStarted","Data":"7bc1ea7b6f1295d0a79141dd760f9fa92bd3c42bd9892651aec4d7b8b090bc04"}
Oct 30 20:06:16.196257 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.195661    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw" event={"ID":"154e4c02-a2b9-4dce-aec0-57c62527b2c4","Type":"ContainerStarted","Data":"ddb7060e7e4cd50f66c392857d20efcddeea7da2a3901c43ad8b2ec75e809c2f"}
Oct 30 20:06:16.201982 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.201853    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"]
Oct 30 20:06:16.209541 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.209518    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.227211 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.227172    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1f5eee\"/\"openshift-service-ca.crt\""
Oct 30 20:06:16.227374 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.227318    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1f5eee\"/\"kube-root-ca.crt\""
Oct 30 20:06:16.227477 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.227457    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1f5eee\"/\"test-model-1f5eee-kserve-self-signed-certs\""
Oct 30 20:06:16.228522 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.228483    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1f5eee\"/\"default-dockercfg-csts9\""
Oct 30 20:06:16.230524 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.230502    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"]
Oct 30 20:06:16.233484 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.233443    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw" podStartSLOduration=2.233431793 podStartE2EDuration="2.233431793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:16.230535893 +0000 UTC m=+2130.814805511" watchObservedRunningTime="2026-10-30 20:06:16.233431793 +0000 UTC m=+2130.817701410"
Oct 30 20:06:16.332165 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.332129    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-home\") pod \"test-model-1f5eee-kserve-7c46b74789-mr4vr\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") " pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.332363 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.332279    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-model-cache\") pod \"test-model-1f5eee-kserve-7c46b74789-mr4vr\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") " pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.332447 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.332393    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-dshm\") pod \"test-model-1f5eee-kserve-7c46b74789-mr4vr\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") " pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.332547 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.332530    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tqx2\" (UniqueName: \"kubernetes.io/projected/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-kube-api-access-6tqx2\") pod \"test-model-1f5eee-kserve-7c46b74789-mr4vr\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") " pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.332685 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.332671    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-tmp-dir\") pod \"test-model-1f5eee-kserve-7c46b74789-mr4vr\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") " pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.332754 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.332733    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-tls-certs\") pod \"test-model-1f5eee-kserve-7c46b74789-mr4vr\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") " pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.436780 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.436741    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-home\") pod \"test-model-1f5eee-kserve-7c46b74789-mr4vr\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") " pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.442574 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.442542    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-model-cache\") pod \"test-model-1f5eee-kserve-7c46b74789-mr4vr\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") " pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.442910 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.442887    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-dshm\") pod \"test-model-1f5eee-kserve-7c46b74789-mr4vr\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") " pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.443175 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.443157    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tqx2\" (UniqueName: \"kubernetes.io/projected/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-kube-api-access-6tqx2\") pod \"test-model-1f5eee-kserve-7c46b74789-mr4vr\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") " pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.452097 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.443641    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-tmp-dir\") pod \"test-model-1f5eee-kserve-7c46b74789-mr4vr\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") " pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.493950 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.490392    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-tls-certs\") pod \"test-model-1f5eee-kserve-7c46b74789-mr4vr\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") " pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.493950 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.437699    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-home\") pod \"test-model-1f5eee-kserve-7c46b74789-mr4vr\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") " pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.493950 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.466766    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-tmp-dir\") pod \"test-model-1f5eee-kserve-7c46b74789-mr4vr\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") " pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.493950 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.474399    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-dshm\") pod \"test-model-1f5eee-kserve-7c46b74789-mr4vr\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") " pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.493950 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.454576    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-model-cache\") pod \"test-model-1f5eee-kserve-7c46b74789-mr4vr\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") " pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.525051 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.510622    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-tls-certs\") pod \"test-model-1f5eee-kserve-7c46b74789-mr4vr\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") " pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.525051 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.522962    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tqx2\" (UniqueName: \"kubernetes.io/projected/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-kube-api-access-6tqx2\") pod \"test-model-1f5eee-kserve-7c46b74789-mr4vr\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") " pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.526696 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.526668    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:16.719226 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:16.719117    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:17.104543 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:17.104513    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"]
Oct 30 20:06:17.107946 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:06:17.107920    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea5acb91_6d26_4368_bbf8_a40eaa8d7d24.slice/crio-cd00dc88d1d5aae9270bf2dd86d33ac2318d6a2e94ec43f330383a51bed927a5 WatchSource:0}: Error finding container cd00dc88d1d5aae9270bf2dd86d33ac2318d6a2e94ec43f330383a51bed927a5: Status 404 returned error can't find the container with id cd00dc88d1d5aae9270bf2dd86d33ac2318d6a2e94ec43f330383a51bed927a5
Oct 30 20:06:17.205966 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:17.205940    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr" event={"ID":"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24","Type":"ContainerStarted","Data":"cd00dc88d1d5aae9270bf2dd86d33ac2318d6a2e94ec43f330383a51bed927a5"}
Oct 30 20:06:18.216267 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:18.216230    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr" event={"ID":"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24","Type":"ContainerStarted","Data":"61bbc667a8c050eddc0783fee135dedaec755c31519ea055cdbeea40d7be7464"}
Oct 30 20:06:18.244141 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:18.244089    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr" podStartSLOduration=2.244074477 podStartE2EDuration="2.244074477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:18.23840072 +0000 UTC m=+2132.822670320" watchObservedRunningTime="2026-10-30 20:06:18.244074477 +0000 UTC m=+2132.828344093"
Oct 30 20:06:18.527214 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:18.527176    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:21.924260 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:21.924228    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"]
Oct 30 20:06:21.939113 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:21.939082    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"]
Oct 30 20:06:21.939302 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:21.939180    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:21.951819 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:21.951792    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-55d2e0\"/\"test-model-55d2e0-kserve-self-signed-certs\""
Oct 30 20:06:21.952161 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:21.952137    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-55d2e0\"/\"openshift-service-ca.crt\""
Oct 30 20:06:21.952226 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:21.952201    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-55d2e0\"/\"default-dockercfg-x25ts\""
Oct 30 20:06:21.952289 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:21.952276    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-55d2e0\"/\"kube-root-ca.crt\""
Oct 30 20:06:22.034255 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.034214    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-dshm\") pod \"test-model-55d2e0-kserve-59bb5dfcb5-tn6b7\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") " pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.034472 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.034433    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-model-cache\") pod \"test-model-55d2e0-kserve-59bb5dfcb5-tn6b7\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") " pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.034538 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.034485    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-tmp-dir\") pod \"test-model-55d2e0-kserve-59bb5dfcb5-tn6b7\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") " pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.034674 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.034657    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w98dm\" (UniqueName: \"kubernetes.io/projected/ac13574a-40f3-4228-affd-51573e0296c1-kube-api-access-w98dm\") pod \"test-model-55d2e0-kserve-59bb5dfcb5-tn6b7\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") " pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.034733 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.034720    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac13574a-40f3-4228-affd-51573e0296c1-tls-certs\") pod \"test-model-55d2e0-kserve-59bb5dfcb5-tn6b7\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") " pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.034796 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.034782    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-home\") pod \"test-model-55d2e0-kserve-59bb5dfcb5-tn6b7\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") " pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.137011 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.136971    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-dshm\") pod \"test-model-55d2e0-kserve-59bb5dfcb5-tn6b7\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") " pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.137548 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.137506    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-model-cache\") pod \"test-model-55d2e0-kserve-59bb5dfcb5-tn6b7\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") " pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.137840 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.137797    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-tmp-dir\") pod \"test-model-55d2e0-kserve-59bb5dfcb5-tn6b7\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") " pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.138155 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.138131    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w98dm\" (UniqueName: \"kubernetes.io/projected/ac13574a-40f3-4228-affd-51573e0296c1-kube-api-access-w98dm\") pod \"test-model-55d2e0-kserve-59bb5dfcb5-tn6b7\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") " pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.138341 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.138324    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac13574a-40f3-4228-affd-51573e0296c1-tls-certs\") pod \"test-model-55d2e0-kserve-59bb5dfcb5-tn6b7\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") " pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.138504 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.138486    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-home\") pod \"test-model-55d2e0-kserve-59bb5dfcb5-tn6b7\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") " pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.139544 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.139523    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-home\") pod \"test-model-55d2e0-kserve-59bb5dfcb5-tn6b7\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") " pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.159431 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.159397    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-tmp-dir\") pod \"test-model-55d2e0-kserve-59bb5dfcb5-tn6b7\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") " pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.160062 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.159974    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-model-cache\") pod \"test-model-55d2e0-kserve-59bb5dfcb5-tn6b7\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") " pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.183109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.169711    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-dshm\") pod \"test-model-55d2e0-kserve-59bb5dfcb5-tn6b7\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") " pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.191065 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.185424    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac13574a-40f3-4228-affd-51573e0296c1-tls-certs\") pod \"test-model-55d2e0-kserve-59bb5dfcb5-tn6b7\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") " pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.216052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.214797    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w98dm\" (UniqueName: \"kubernetes.io/projected/ac13574a-40f3-4228-affd-51573e0296c1-kube-api-access-w98dm\") pod \"test-model-55d2e0-kserve-59bb5dfcb5-tn6b7\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") " pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.308492 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.299392    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:22.812546 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:22.811733    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"]
Oct 30 20:06:22.814966 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:06:22.814938    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac13574a_40f3_4228_affd_51573e0296c1.slice/crio-33421ed289b23f1a6b9658ebc50c4f073343d00b2bffe79d1988989c51c83e9a WatchSource:0}: Error finding container 33421ed289b23f1a6b9658ebc50c4f073343d00b2bffe79d1988989c51c83e9a: Status 404 returned error can't find the container with id 33421ed289b23f1a6b9658ebc50c4f073343d00b2bffe79d1988989c51c83e9a
Oct 30 20:06:23.297423 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:23.297381    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7" event={"ID":"ac13574a-40f3-4228-affd-51573e0296c1","Type":"ContainerStarted","Data":"7c4ba5eecef01a5068afbb5515bcc7d89c5389ba122b6dfa707df5b0ecedd05a"}
Oct 30 20:06:23.298008 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:23.297969    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7" event={"ID":"ac13574a-40f3-4228-affd-51573e0296c1","Type":"ContainerStarted","Data":"33421ed289b23f1a6b9658ebc50c4f073343d00b2bffe79d1988989c51c83e9a"}
Oct 30 20:06:23.302919 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:23.302892    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:23.326425 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:23.326305    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7" podStartSLOduration=2.326286912 podStartE2EDuration="2.326286912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:23.319477122 +0000 UTC m=+2137.903746738" watchObservedRunningTime="2026-10-30 20:06:23.326286912 +0000 UTC m=+2137.910556530"
Oct 30 20:06:24.715930 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:24.715890    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:24.728900 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:24.728866    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:25.333354 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:25.333330    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:06:26.060815 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.060772    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"]
Oct 30 20:06:26.066285 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.066262    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.084291 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.084260    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"]
Oct 30 20:06:26.084825 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.084806    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7f7d21\"/\"default-dockercfg-dsmtb\""
Oct 30 20:06:26.086237 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.086216    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7f7d21\"/\"test-model-7f7d21-kserve-self-signed-certs\""
Oct 30 20:06:26.140997 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.140963    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7f7d21\"/\"kube-root-ca.crt\""
Oct 30 20:06:26.141291 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.141270    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7f7d21\"/\"openshift-service-ca.crt\""
Oct 30 20:06:26.169560 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.169528    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-dshm\") pod \"test-model-7f7d21-kserve-594d95cbf7-l2ntn\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") " pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.169738 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.169575    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dd5e806-31e7-4ae8-90d3-7806662cfd16-tls-certs\") pod \"test-model-7f7d21-kserve-594d95cbf7-l2ntn\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") " pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.169738 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.169717    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-home\") pod \"test-model-7f7d21-kserve-594d95cbf7-l2ntn\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") " pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.169849 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.169810    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kv6g\" (UniqueName: \"kubernetes.io/projected/0dd5e806-31e7-4ae8-90d3-7806662cfd16-kube-api-access-4kv6g\") pod \"test-model-7f7d21-kserve-594d95cbf7-l2ntn\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") " pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.169975 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.169956    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-model-cache\") pod \"test-model-7f7d21-kserve-594d95cbf7-l2ntn\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") " pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.170128 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.170111    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-tmp-dir\") pod \"test-model-7f7d21-kserve-594d95cbf7-l2ntn\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") " pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.274061 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.273672    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-dshm\") pod \"test-model-7f7d21-kserve-594d95cbf7-l2ntn\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") " pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.274061 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.273733    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dd5e806-31e7-4ae8-90d3-7806662cfd16-tls-certs\") pod \"test-model-7f7d21-kserve-594d95cbf7-l2ntn\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") " pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.274061 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.273802    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-home\") pod \"test-model-7f7d21-kserve-594d95cbf7-l2ntn\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") " pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.274061 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.273885    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kv6g\" (UniqueName: \"kubernetes.io/projected/0dd5e806-31e7-4ae8-90d3-7806662cfd16-kube-api-access-4kv6g\") pod \"test-model-7f7d21-kserve-594d95cbf7-l2ntn\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") " pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.274061 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.273993    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-model-cache\") pod \"test-model-7f7d21-kserve-594d95cbf7-l2ntn\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") " pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.274722 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.274132    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-tmp-dir\") pod \"test-model-7f7d21-kserve-594d95cbf7-l2ntn\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") " pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.275217 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.274839    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-tmp-dir\") pod \"test-model-7f7d21-kserve-594d95cbf7-l2ntn\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") " pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.293049 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.283089    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-model-cache\") pod \"test-model-7f7d21-kserve-594d95cbf7-l2ntn\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") " pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.293049 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.283364    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-home\") pod \"test-model-7f7d21-kserve-594d95cbf7-l2ntn\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") " pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.310086 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.302743    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dd5e806-31e7-4ae8-90d3-7806662cfd16-tls-certs\") pod \"test-model-7f7d21-kserve-594d95cbf7-l2ntn\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") " pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.324119 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.322635    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-dshm\") pod \"test-model-7f7d21-kserve-594d95cbf7-l2ntn\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") " pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.341134 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.336000    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kv6g\" (UniqueName: \"kubernetes.io/projected/0dd5e806-31e7-4ae8-90d3-7806662cfd16-kube-api-access-4kv6g\") pod \"test-model-7f7d21-kserve-594d95cbf7-l2ntn\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") " pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.411054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.408563    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:26.559083 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.554347    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:26.615678 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:26.611711    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:27.015910 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:06:27.015877    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dd5e806_31e7_4ae8_90d3_7806662cfd16.slice/crio-53c53eff776f41c4b12adfb91472dc95a9b156d5e85114cf6fae8a832fe3b542 WatchSource:0}: Error finding container 53c53eff776f41c4b12adfb91472dc95a9b156d5e85114cf6fae8a832fe3b542: Status 404 returned error can't find the container with id 53c53eff776f41c4b12adfb91472dc95a9b156d5e85114cf6fae8a832fe3b542
Oct 30 20:06:27.017621 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:27.017592    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"]
Oct 30 20:06:27.440867 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:27.440775    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn" event={"ID":"0dd5e806-31e7-4ae8-90d3-7806662cfd16","Type":"ContainerStarted","Data":"87612665866e9be3f6d6967c4bd812adb99570aa39dbf99d720a36e2cbff4fb9"}
Oct 30 20:06:27.440867 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:27.440822    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn" event={"ID":"0dd5e806-31e7-4ae8-90d3-7806662cfd16","Type":"ContainerStarted","Data":"53c53eff776f41c4b12adfb91472dc95a9b156d5e85114cf6fae8a832fe3b542"}
Oct 30 20:06:27.453118 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:27.453091    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:06:27.467523 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:27.467457    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn" podStartSLOduration=1.467433373 podStartE2EDuration="1.467433373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:27.463054421 +0000 UTC m=+2142.047324037" watchObservedRunningTime="2026-10-30 20:06:27.467433373 +0000 UTC m=+2142.051702980"
Oct 30 20:06:28.420881 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:28.420849    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:32.303796 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:32.303757    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:32.317393 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:32.317363    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:32.496824 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:32.496791    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:06:36.420692 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:36.420651    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:36.433595 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:36.433568    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:36.531419 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:36.531390    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:06:39.662484 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:06:39.662450    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:06:41.355066 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.354992    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"]
Oct 30 20:06:41.360070 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.360051    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.374660 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.374637    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6986a4\"/\"test-model-6986a4-kserve-self-signed-certs\""
Oct 30 20:06:41.375691 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.375668    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6986a4\"/\"default-dockercfg-drmhc\""
Oct 30 20:06:41.375834 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.375715    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6986a4\"/\"openshift-service-ca.crt\""
Oct 30 20:06:41.375834 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.375720    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6986a4\"/\"kube-root-ca.crt\""
Oct 30 20:06:41.414010 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.413986    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"]
Oct 30 20:06:41.536748 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.536707    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-tmp-dir\") pod \"test-model-6986a4-kserve-7b9b7c5567-n27vz\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") " pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.536945 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.536769    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-dshm\") pod \"test-model-6986a4-kserve-7b9b7c5567-n27vz\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") " pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.536945 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.536894    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qklct\" (UniqueName: \"kubernetes.io/projected/cc5cea2d-0de3-445f-923c-44341da06bc8-kube-api-access-qklct\") pod \"test-model-6986a4-kserve-7b9b7c5567-n27vz\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") " pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.537140 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.537118    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc5cea2d-0de3-445f-923c-44341da06bc8-tls-certs\") pod \"test-model-6986a4-kserve-7b9b7c5567-n27vz\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") " pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.537208 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.537193    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-model-cache\") pod \"test-model-6986a4-kserve-7b9b7c5567-n27vz\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") " pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.537287 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.537254    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-home\") pod \"test-model-6986a4-kserve-7b9b7c5567-n27vz\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") " pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.652099 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.640340    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-tmp-dir\") pod \"test-model-6986a4-kserve-7b9b7c5567-n27vz\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") " pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.652099 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.640408    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-dshm\") pod \"test-model-6986a4-kserve-7b9b7c5567-n27vz\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") " pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.652099 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.640516    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qklct\" (UniqueName: \"kubernetes.io/projected/cc5cea2d-0de3-445f-923c-44341da06bc8-kube-api-access-qklct\") pod \"test-model-6986a4-kserve-7b9b7c5567-n27vz\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") " pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.652099 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.640738    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc5cea2d-0de3-445f-923c-44341da06bc8-tls-certs\") pod \"test-model-6986a4-kserve-7b9b7c5567-n27vz\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") " pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.652099 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.640811    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-model-cache\") pod \"test-model-6986a4-kserve-7b9b7c5567-n27vz\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") " pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.652099 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.640868    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-home\") pod \"test-model-6986a4-kserve-7b9b7c5567-n27vz\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") " pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.652099 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.641470    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-home\") pod \"test-model-6986a4-kserve-7b9b7c5567-n27vz\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") " pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.652099 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.641864    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-tmp-dir\") pod \"test-model-6986a4-kserve-7b9b7c5567-n27vz\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") " pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.652099 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.642877    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-model-cache\") pod \"test-model-6986a4-kserve-7b9b7c5567-n27vz\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") " pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.652099 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.645517    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-dshm\") pod \"test-model-6986a4-kserve-7b9b7c5567-n27vz\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") " pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.685079 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.683022    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc5cea2d-0de3-445f-923c-44341da06bc8-tls-certs\") pod \"test-model-6986a4-kserve-7b9b7c5567-n27vz\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") " pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.709050 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.689460    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qklct\" (UniqueName: \"kubernetes.io/projected/cc5cea2d-0de3-445f-923c-44341da06bc8-kube-api-access-qklct\") pod \"test-model-6986a4-kserve-7b9b7c5567-n27vz\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") " pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:41.976142 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:41.976063    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:42.278384 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:42.278352    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"]
Oct 30 20:06:42.282459 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:06:42.282428    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc5cea2d_0de3_445f_923c_44341da06bc8.slice/crio-3ac7ebcd0ce8aae4e8b06ab3fad16693d545cf43152d564ab99d50e242b51a3e WatchSource:0}: Error finding container 3ac7ebcd0ce8aae4e8b06ab3fad16693d545cf43152d564ab99d50e242b51a3e: Status 404 returned error can't find the container with id 3ac7ebcd0ce8aae4e8b06ab3fad16693d545cf43152d564ab99d50e242b51a3e
Oct 30 20:06:42.583756 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:42.583676    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz" event={"ID":"cc5cea2d-0de3-445f-923c-44341da06bc8","Type":"ContainerStarted","Data":"b08475cfcb3149f00d87dcd7a52dc6df71193984c118ed753332b51f80be1b21"}
Oct 30 20:06:42.583756 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:42.583725    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz" event={"ID":"cc5cea2d-0de3-445f-923c-44341da06bc8","Type":"ContainerStarted","Data":"3ac7ebcd0ce8aae4e8b06ab3fad16693d545cf43152d564ab99d50e242b51a3e"}
Oct 30 20:06:42.615070 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:42.612994    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz" podStartSLOduration=1.6129790750000002 podStartE2EDuration="1.612979075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:42.607236183 +0000 UTC m=+2157.191505794" watchObservedRunningTime="2026-10-30 20:06:42.612979075 +0000 UTC m=+2157.197248691"
Oct 30 20:06:42.976439 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:42.976365    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:51.976446 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:51.976414    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:51.989712 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:51.989685    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:52.977647 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:52.977614    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:06:54.872811 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:54.871873    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2" podUID="d3cbd0cd-db30-4181-9271-fab50895d832" containerName="main" probeResult="failure" output="Get \"https://10.134.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:54.969183 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:54.969135    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt" podUID="90d486b1-ecce-40a9-8353-04127a5111df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.009167 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.009124    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd" podUID="b19efa61-42ee-40ce-98b4-6b57dd46b3c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.155:8000/ready\": context deadline exceeded"
Oct 30 20:06:55.057933 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.057888    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 20:06:55.108171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.108122    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf" podUID="8300c88b-76da-4c18-b214-49d9ae483bd7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.147106 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.146902    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh" podUID="933df3f9-ffba-40a6-b455-4ddf13776704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.150:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.168674 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.168631    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29" podUID="3558d905-d0a3-4b3f-8c5c-dac3353f4b1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.156:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.402786 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.401169    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw" podUID="154e4c02-a2b9-4dce-aec0-57c62527b2c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.452375 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.451243    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 20:06:55.473480 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.472461    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr" podUID="ea5acb91-6d26-4368-bbf8-a40eaa8d7d24" containerName="main" probeResult="failure" output="Get \"https://10.134.0.162:8000/ready\": context deadline exceeded"
Oct 30 20:06:55.487180 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.487121    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd" podUID="b19efa61-42ee-40ce-98b4-6b57dd46b3c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.155:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.528980 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.528936    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn" podUID="0dd5e806-31e7-4ae8-90d3-7806662cfd16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.164:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.586395 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.585615    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.624120 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.624024    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.627275 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.627239    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.712717 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.712518    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb" podUID="9b56dd57-5a11-4834-a128-5aa9e2385157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.739685 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.739629    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.759289 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.759237    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls" podUID="f8b220f9-8403-4d34-a940-200f160de17f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.760582 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.760548    5408 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw" podUID="154e4c02-a2b9-4dce-aec0-57c62527b2c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.161:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.760868 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.760843    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz" podUID="cc5cea2d-0de3-445f-923c-44341da06bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.165:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.970385 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.970280    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt" podUID="90d486b1-ecce-40a9-8353-04127a5111df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded"
Oct 30 20:06:55.976064 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:55.971161    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:06:56.059398 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:06:56.056239    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:07:09.696483 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:07:09.696388    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:07:28.180945 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.180912    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"]
Oct 30 20:07:28.186320 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.186299    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.191256 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.191236    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f246b-openshift-default-dockercfg-nb8cn\""
Oct 30 20:07:28.198288 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.198261    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"]
Oct 30 20:07:28.351372 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.351336    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-envoy\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.351542 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.351458    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-workload-certs\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.351542 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.351534    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-credential-socket\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.351683 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.351606    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-podinfo\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.351719 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.351695    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44f7z\" (UniqueName: \"kubernetes.io/projected/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-kube-api-access-44f7z\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.351754 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.351735    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-token\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.351803 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.351791    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-workload-socket\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.351892 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.351878    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-data\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.351929 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.351899    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istiod-ca-cert\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.453083 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.452964    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-data\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.453083 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.453004    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istiod-ca-cert\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.453299 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.453109    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-envoy\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.453299 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.453184    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-workload-certs\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.453397 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.453307    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-credential-socket\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.453397 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.453355    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-podinfo\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.453504 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.453435    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-data\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.453504 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.453469    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44f7z\" (UniqueName: \"kubernetes.io/projected/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-kube-api-access-44f7z\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.453504 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.453499    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-token\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.453671 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.453517    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-workload-certs\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.453671 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.453591    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-workload-socket\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.453821 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.453795    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istiod-ca-cert\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.453949 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.453932    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-workload-socket\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.454139 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.454114    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-credential-socket\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.455854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.455829    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-podinfo\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.455958 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.455859    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-envoy\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.467100 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.467077    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-token\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.467207 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.467162    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44f7z\" (UniqueName: \"kubernetes.io/projected/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-kube-api-access-44f7z\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.507337 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.507307    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:28.854281 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.853920    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"]
Oct 30 20:07:28.871681 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.871637    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:07:28.871812 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.871729    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:07:28.871812 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:28.871773    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:07:29.342458 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:29.342425    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt" event={"ID":"193b7fcd-4f9f-43d4-b833-e8f690bf5df8","Type":"ContainerStarted","Data":"982f4b5b99705fe79e75a763604796f3ce9ddc4ddcb0f27435bb4f243607c9a9"}
Oct 30 20:07:29.342930 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:29.342506    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt" event={"ID":"193b7fcd-4f9f-43d4-b833-e8f690bf5df8","Type":"ContainerStarted","Data":"153251e63c7e21e4191f2eb8fddb5c9e9a2fe44656e6e9d681d832df3c39500a"}
Oct 30 20:07:29.379925 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:29.379878    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt" podStartSLOduration=1.379864519 podStartE2EDuration="1.379864519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:29.376827332 +0000 UTC m=+2203.961096950" watchObservedRunningTime="2026-10-30 20:07:29.379864519 +0000 UTC m=+2203.964134136"
Oct 30 20:07:29.512051 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:29.511388    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:29.522915 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:29.521213    5408 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 30 20:07:29.522915 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:29.521288    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt" podUID="193b7fcd-4f9f-43d4-b833-e8f690bf5df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 30 20:07:30.523079 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:30.515621    5408 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 30 20:07:30.523079 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:30.515681    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt" podUID="193b7fcd-4f9f-43d4-b833-e8f690bf5df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 30 20:07:31.513053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:31.510285    5408 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 30 20:07:31.513053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:31.510357    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt" podUID="193b7fcd-4f9f-43d4-b833-e8f690bf5df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 30 20:07:32.515053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:32.513655    5408 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 30 20:07:32.515053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:32.513717    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt" podUID="193b7fcd-4f9f-43d4-b833-e8f690bf5df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 30 20:07:33.525055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:33.522490    5408 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 30 20:07:33.525055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:33.522547    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt" podUID="193b7fcd-4f9f-43d4-b833-e8f690bf5df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 30 20:07:34.514059 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:34.510578    5408 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 30 20:07:34.514059 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:34.510653    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt" podUID="193b7fcd-4f9f-43d4-b833-e8f690bf5df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 30 20:07:35.620332 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:35.620300    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:35.621669 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:35.621641    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:35.626300 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:35.626278    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:07:39.663728 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:07:39.663694    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:07:53.713063 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:53.696954    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:53.742311 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:53.742262    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls" podUID="f8b220f9-8403-4d34-a940-200f160de17f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:53.915247 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:53.713800    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb" podUID="9b56dd57-5a11-4834-a128-5aa9e2385157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:53.915547 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:53.743210    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 20:07:53.915837 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:53.882481    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2" podUID="d3cbd0cd-db30-4181-9271-fab50895d832" containerName="main" probeResult="failure" output="Get \"https://10.134.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:54.060253 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:54.060196    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt" podUID="90d486b1-ecce-40a9-8353-04127a5111df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:54.097258 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:54.097206    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf" podUID="8300c88b-76da-4c18-b214-49d9ae483bd7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.160:8000/ready\": context deadline exceeded"
Oct 30 20:07:54.150703 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:07:54.150660    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:09.662421 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:08:09.662384    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:08:39.662463 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:08:39.662370    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:08:55.606795 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:08:55.606762    5408 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kspfd_23de5b1c-4825-43ed-bff8-cdab8789989e/konnectivity-agent/0.log" limit=1024
Oct 30 20:09:06.209160 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:06.209123    5408 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kspfd_23de5b1c-4825-43ed-bff8-cdab8789989e/konnectivity-agent/0.log" limit=1024
Oct 30 20:09:09.662613 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:09:09.662574    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:09:29.704340 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:29.704305    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"]
Oct 30 20:09:38.149960 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:38.149924    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"]
Oct 30 20:09:38.608497 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:38.608452    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x"]
Oct 30 20:09:38.613985 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:38.613962    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x"
Oct 30 20:09:38.625462 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:38.624287    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:09:38.625462 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:38.624430    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:09:38.628396 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:38.628370    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x"]
Oct 30 20:09:38.711005 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:38.710961    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z57dd\" (UniqueName: \"kubernetes.io/projected/f46abb11-ef94-4122-aab3-d1feaaa46d58-kube-api-access-z57dd\") pod \"maas-api-revoke-keys-e2e-inf-a-925e04-2697x\" (UID: \"f46abb11-ef94-4122-aab3-d1feaaa46d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x"
Oct 30 20:09:38.711245 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:38.711107    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46abb11-ef94-4122-aab3-d1feaaa46d58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-925e04-2697x\" (UID: \"f46abb11-ef94-4122-aab3-d1feaaa46d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x"
Oct 30 20:09:38.711329 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:38.711284    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46abb11-ef94-4122-aab3-d1feaaa46d58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-925e04-2697x\" (UID: \"f46abb11-ef94-4122-aab3-d1feaaa46d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x"
Oct 30 20:09:38.812665 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:38.812629    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46abb11-ef94-4122-aab3-d1feaaa46d58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-925e04-2697x\" (UID: \"f46abb11-ef94-4122-aab3-d1feaaa46d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x"
Oct 30 20:09:38.812955 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:38.812931    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z57dd\" (UniqueName: \"kubernetes.io/projected/f46abb11-ef94-4122-aab3-d1feaaa46d58-kube-api-access-z57dd\") pod \"maas-api-revoke-keys-e2e-inf-a-925e04-2697x\" (UID: \"f46abb11-ef94-4122-aab3-d1feaaa46d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x"
Oct 30 20:09:38.813094 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:38.813076    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46abb11-ef94-4122-aab3-d1feaaa46d58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-925e04-2697x\" (UID: \"f46abb11-ef94-4122-aab3-d1feaaa46d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x"
Oct 30 20:09:38.813467 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:38.813447    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46abb11-ef94-4122-aab3-d1feaaa46d58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-925e04-2697x\" (UID: \"f46abb11-ef94-4122-aab3-d1feaaa46d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x"
Oct 30 20:09:38.815687 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:38.815664    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46abb11-ef94-4122-aab3-d1feaaa46d58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-925e04-2697x\" (UID: \"f46abb11-ef94-4122-aab3-d1feaaa46d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x"
Oct 30 20:09:38.834266 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:38.834237    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z57dd\" (UniqueName: \"kubernetes.io/projected/f46abb11-ef94-4122-aab3-d1feaaa46d58-kube-api-access-z57dd\") pod \"maas-api-revoke-keys-e2e-inf-a-925e04-2697x\" (UID: \"f46abb11-ef94-4122-aab3-d1feaaa46d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x"
Oct 30 20:09:38.938636 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:38.938563    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x"
Oct 30 20:09:39.240628 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:39.240601    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x"]
Oct 30 20:09:39.242369 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:09:39.242331    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46abb11_ef94_4122_aab3_d1feaaa46d58.slice/crio-86ded9e6e36989840580ed5a6596e621955490d1a2609131ed5a0df84f3b4575 WatchSource:0}: Error finding container 86ded9e6e36989840580ed5a6596e621955490d1a2609131ed5a0df84f3b4575: Status 404 returned error can't find the container with id 86ded9e6e36989840580ed5a6596e621955490d1a2609131ed5a0df84f3b4575
Oct 30 20:09:39.662780 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:09:39.662708    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:09:40.252046 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:40.252006    5408 generic.go:356] "Generic (PLEG): container finished" podID="f46abb11-ef94-4122-aab3-d1feaaa46d58" containerID="b9a85ce8a978526993769918c905c099959169fba8327f31a3a5349550468d76" exitCode=0
Oct 30 20:09:40.252398 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:40.252073    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x" event={"ID":"f46abb11-ef94-4122-aab3-d1feaaa46d58","Type":"ContainerDied","Data":"b9a85ce8a978526993769918c905c099959169fba8327f31a3a5349550468d76"}
Oct 30 20:09:40.252398 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:40.252135    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x" event={"ID":"f46abb11-ef94-4122-aab3-d1feaaa46d58","Type":"ContainerStarted","Data":"86ded9e6e36989840580ed5a6596e621955490d1a2609131ed5a0df84f3b4575"}
Oct 30 20:09:41.439652 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:41.439626    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x"
Oct 30 20:09:41.481144 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:41.481106    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46abb11-ef94-4122-aab3-d1feaaa46d58-kube-api-access-z57dd\" (UniqueName: \"kubernetes.io/projected/f46abb11-ef94-4122-aab3-d1feaaa46d58-kube-api-access-z57dd\") pod \"f46abb11-ef94-4122-aab3-d1feaaa46d58\" (UID: \"f46abb11-ef94-4122-aab3-d1feaaa46d58\") "
Oct 30 20:09:41.481343 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:41.481281    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46abb11-ef94-4122-aab3-d1feaaa46d58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46abb11-ef94-4122-aab3-d1feaaa46d58-maas-api-openshift-ca\") pod \"f46abb11-ef94-4122-aab3-d1feaaa46d58\" (UID: \"f46abb11-ef94-4122-aab3-d1feaaa46d58\") "
Oct 30 20:09:41.481343 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:41.481315    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46abb11-ef94-4122-aab3-d1feaaa46d58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46abb11-ef94-4122-aab3-d1feaaa46d58-maas-api-xks-ca\") pod \"f46abb11-ef94-4122-aab3-d1feaaa46d58\" (UID: \"f46abb11-ef94-4122-aab3-d1feaaa46d58\") "
Oct 30 20:09:41.481711 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:41.481685    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46abb11-ef94-4122-aab3-d1feaaa46d58-maas-api-openshift-ca" pod "f46abb11-ef94-4122-aab3-d1feaaa46d58" (UID: "f46abb11-ef94-4122-aab3-d1feaaa46d58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:09:41.482455 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:41.482430    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46abb11-ef94-4122-aab3-d1feaaa46d58-maas-api-openshift-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:41.483872 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:41.483848    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46abb11-ef94-4122-aab3-d1feaaa46d58-maas-api-xks-ca" pod "f46abb11-ef94-4122-aab3-d1feaaa46d58" (UID: "f46abb11-ef94-4122-aab3-d1feaaa46d58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:09:41.483981 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:41.483962    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46abb11-ef94-4122-aab3-d1feaaa46d58-kube-api-access-z57dd" pod "f46abb11-ef94-4122-aab3-d1feaaa46d58" (UID: "f46abb11-ef94-4122-aab3-d1feaaa46d58"). InnerVolumeSpecName "kube-api-access-z57dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:41.585052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:41.585004    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z57dd\" (UniqueName: \"kubernetes.io/projected/f46abb11-ef94-4122-aab3-d1feaaa46d58-kube-api-access-z57dd\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:41.585052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:41.585048    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46abb11-ef94-4122-aab3-d1feaaa46d58-maas-api-xks-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:42.270427 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:42.270391    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x" event={"ID":"f46abb11-ef94-4122-aab3-d1feaaa46d58","Type":"ContainerDied","Data":"86ded9e6e36989840580ed5a6596e621955490d1a2609131ed5a0df84f3b4575"}
Oct 30 20:09:42.270427 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:42.270436    5408 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86ded9e6e36989840580ed5a6596e621955490d1a2609131ed5a0df84f3b4575"
Oct 30 20:09:42.270678 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:42.270397    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x"
Oct 30 20:09:44.731950 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:44.731904    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf" podUID="8300c88b-76da-4c18-b214-49d9ae483bd7" containerName="main" containerID="cri-o://371d82f04ce43bb10c457a6a1c3354d459d60d3bc4f9fd8ea45db9a92195d917" gracePeriod=45
Oct 30 20:09:45.060953 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.060926    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:09:45.160233 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.160202    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-dshm\") pod \"8300c88b-76da-4c18-b214-49d9ae483bd7\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") "
Oct 30 20:09:45.160425 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.160309    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8300c88b-76da-4c18-b214-49d9ae483bd7-kube-api-access-hqr72\" (UniqueName: \"kubernetes.io/projected/8300c88b-76da-4c18-b214-49d9ae483bd7-kube-api-access-hqr72\") pod \"8300c88b-76da-4c18-b214-49d9ae483bd7\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") "
Oct 30 20:09:45.160505 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.160489    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-model-cache\") pod \"8300c88b-76da-4c18-b214-49d9ae483bd7\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") "
Oct 30 20:09:45.160620 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.160606    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-tmp-dir\") pod \"8300c88b-76da-4c18-b214-49d9ae483bd7\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") "
Oct 30 20:09:45.160694 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.160682    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-home\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-home\") pod \"8300c88b-76da-4c18-b214-49d9ae483bd7\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") "
Oct 30 20:09:45.160751 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.160718    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8300c88b-76da-4c18-b214-49d9ae483bd7-tls-certs\" (UniqueName: \"kubernetes.io/secret/8300c88b-76da-4c18-b214-49d9ae483bd7-tls-certs\") pod \"8300c88b-76da-4c18-b214-49d9ae483bd7\" (UID: \"8300c88b-76da-4c18-b214-49d9ae483bd7\") "
Oct 30 20:09:45.160804 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.160746    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-model-cache" pod "8300c88b-76da-4c18-b214-49d9ae483bd7" (UID: "8300c88b-76da-4c18-b214-49d9ae483bd7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:45.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.160872    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-tmp-dir" pod "8300c88b-76da-4c18-b214-49d9ae483bd7" (UID: "8300c88b-76da-4c18-b214-49d9ae483bd7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:45.160960 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.160936    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-home" pod "8300c88b-76da-4c18-b214-49d9ae483bd7" (UID: "8300c88b-76da-4c18-b214-49d9ae483bd7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:45.161510 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.161482    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:45.161510 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.161503    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:45.161681 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.161516    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:45.163904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.163868    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8300c88b-76da-4c18-b214-49d9ae483bd7-tls-certs" pod "8300c88b-76da-4c18-b214-49d9ae483bd7" (UID: "8300c88b-76da-4c18-b214-49d9ae483bd7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:09:45.163904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.163881    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8300c88b-76da-4c18-b214-49d9ae483bd7-kube-api-access-hqr72" pod "8300c88b-76da-4c18-b214-49d9ae483bd7" (UID: "8300c88b-76da-4c18-b214-49d9ae483bd7"). InnerVolumeSpecName "kube-api-access-hqr72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:45.166162 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.166139    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-dshm" pod "8300c88b-76da-4c18-b214-49d9ae483bd7" (UID: "8300c88b-76da-4c18-b214-49d9ae483bd7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:45.263337 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.263310    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8300c88b-76da-4c18-b214-49d9ae483bd7-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:45.263337 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.263334    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8300c88b-76da-4c18-b214-49d9ae483bd7-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:45.263337 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.263344    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqr72\" (UniqueName: \"kubernetes.io/projected/8300c88b-76da-4c18-b214-49d9ae483bd7-kube-api-access-hqr72\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:45.295284 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.295210    5408 generic.go:356] "Generic (PLEG): container finished" podID="8300c88b-76da-4c18-b214-49d9ae483bd7" containerID="371d82f04ce43bb10c457a6a1c3354d459d60d3bc4f9fd8ea45db9a92195d917" exitCode=0
Oct 30 20:09:45.295284 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.295253    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"
Oct 30 20:09:45.295450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.295301    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf" event={"ID":"8300c88b-76da-4c18-b214-49d9ae483bd7","Type":"ContainerDied","Data":"371d82f04ce43bb10c457a6a1c3354d459d60d3bc4f9fd8ea45db9a92195d917"}
Oct 30 20:09:45.295450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.295354    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf" event={"ID":"8300c88b-76da-4c18-b214-49d9ae483bd7","Type":"ContainerDied","Data":"a2774fc3b1833930d5bacb154ae336f03a91c51e05b016f1fed6777692d1146d"}
Oct 30 20:09:45.295450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.295372    5408 scope.go:122] "RemoveContainer" containerID="371d82f04ce43bb10c457a6a1c3354d459d60d3bc4f9fd8ea45db9a92195d917"
Oct 30 20:09:45.307519 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.307504    5408 scope.go:122] "RemoveContainer" containerID="371d82f04ce43bb10c457a6a1c3354d459d60d3bc4f9fd8ea45db9a92195d917"
Oct 30 20:09:45.307809 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:09:45.307788    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"371d82f04ce43bb10c457a6a1c3354d459d60d3bc4f9fd8ea45db9a92195d917\": container with ID starting with 371d82f04ce43bb10c457a6a1c3354d459d60d3bc4f9fd8ea45db9a92195d917 not found: ID does not exist" containerID="371d82f04ce43bb10c457a6a1c3354d459d60d3bc4f9fd8ea45db9a92195d917"
Oct 30 20:09:45.307885 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.307814    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"371d82f04ce43bb10c457a6a1c3354d459d60d3bc4f9fd8ea45db9a92195d917"} err="failed to get container status \"371d82f04ce43bb10c457a6a1c3354d459d60d3bc4f9fd8ea45db9a92195d917\": rpc error: code = NotFound desc = could not find container \"371d82f04ce43bb10c457a6a1c3354d459d60d3bc4f9fd8ea45db9a92195d917\": container with ID starting with 371d82f04ce43bb10c457a6a1c3354d459d60d3bc4f9fd8ea45db9a92195d917 not found: ID does not exist"
Oct 30 20:09:45.325351 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.325327    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"]
Oct 30 20:09:45.329669 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.329647    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-925e04/test-model-925e04-kserve-7969d4dd9c-hjbwf"]
Oct 30 20:09:45.536312 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:45.536281    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8300c88b-76da-4c18-b214-49d9ae483bd7" path="/var/lib/kubelet/pods/8300c88b-76da-4c18-b214-49d9ae483bd7/volumes"
Oct 30 20:09:53.216724 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.216682    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw" podUID="154e4c02-a2b9-4dce-aec0-57c62527b2c4" containerName="main" containerID="cri-o://7bc1ea7b6f1295d0a79141dd760f9fa92bd3c42bd9892651aec4d7b8b090bc04" gracePeriod=45
Oct 30 20:09:53.322112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.322065    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw" podUID="154e4c02-a2b9-4dce-aec0-57c62527b2c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.161:8000/ready\": dial tcp 10.134.0.161:8000: connect: connection refused"
Oct 30 20:09:53.503811 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.502549    5408 generic.go:356] "Generic (PLEG): container finished" podID="154e4c02-a2b9-4dce-aec0-57c62527b2c4" containerID="7bc1ea7b6f1295d0a79141dd760f9fa92bd3c42bd9892651aec4d7b8b090bc04" exitCode=0
Oct 30 20:09:53.503811 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.502746    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw" event={"ID":"154e4c02-a2b9-4dce-aec0-57c62527b2c4","Type":"ContainerDied","Data":"7bc1ea7b6f1295d0a79141dd760f9fa92bd3c42bd9892651aec4d7b8b090bc04"}
Oct 30 20:09:53.604479 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.604452    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:09:53.685013 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.684984    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-model-cache\") pod \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") "
Oct 30 20:09:53.685197 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.685165    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/154e4c02-a2b9-4dce-aec0-57c62527b2c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/154e4c02-a2b9-4dce-aec0-57c62527b2c4-tls-certs\") pod \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") "
Oct 30 20:09:53.685273 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.685235    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-home\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-home\") pod \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") "
Oct 30 20:09:53.685273 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.685242    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-model-cache" pod "154e4c02-a2b9-4dce-aec0-57c62527b2c4" (UID: "154e4c02-a2b9-4dce-aec0-57c62527b2c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:53.685273 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.685263    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-tmp-dir\") pod \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") "
Oct 30 20:09:53.685452 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.685330    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-dshm\") pod \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") "
Oct 30 20:09:53.685452 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.685375    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154e4c02-a2b9-4dce-aec0-57c62527b2c4-kube-api-access-f598s\" (UniqueName: \"kubernetes.io/projected/154e4c02-a2b9-4dce-aec0-57c62527b2c4-kube-api-access-f598s\") pod \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\" (UID: \"154e4c02-a2b9-4dce-aec0-57c62527b2c4\") "
Oct 30 20:09:53.685452 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.685398    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-home" pod "154e4c02-a2b9-4dce-aec0-57c62527b2c4" (UID: "154e4c02-a2b9-4dce-aec0-57c62527b2c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:53.685632 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.685607    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-tmp-dir" pod "154e4c02-a2b9-4dce-aec0-57c62527b2c4" (UID: "154e4c02-a2b9-4dce-aec0-57c62527b2c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:53.686574 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.686553    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:53.686574 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.686572    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:53.686767 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.686583    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:53.687826 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.687805    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-dshm" pod "154e4c02-a2b9-4dce-aec0-57c62527b2c4" (UID: "154e4c02-a2b9-4dce-aec0-57c62527b2c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:09:53.687957 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.687933    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/154e4c02-a2b9-4dce-aec0-57c62527b2c4-tls-certs" pod "154e4c02-a2b9-4dce-aec0-57c62527b2c4" (UID: "154e4c02-a2b9-4dce-aec0-57c62527b2c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:09:53.688251 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.688227    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154e4c02-a2b9-4dce-aec0-57c62527b2c4-kube-api-access-f598s" pod "154e4c02-a2b9-4dce-aec0-57c62527b2c4" (UID: "154e4c02-a2b9-4dce-aec0-57c62527b2c4"). InnerVolumeSpecName "kube-api-access-f598s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:53.788604 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.788526    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154e4c02-a2b9-4dce-aec0-57c62527b2c4-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:53.788604 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.788553    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154e4c02-a2b9-4dce-aec0-57c62527b2c4-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:53.788604 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:53.788563    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f598s\" (UniqueName: \"kubernetes.io/projected/154e4c02-a2b9-4dce-aec0-57c62527b2c4-kube-api-access-f598s\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:09:54.511661 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:54.511627    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw" event={"ID":"154e4c02-a2b9-4dce-aec0-57c62527b2c4","Type":"ContainerDied","Data":"ddb7060e7e4cd50f66c392857d20efcddeea7da2a3901c43ad8b2ec75e809c2f"}
Oct 30 20:09:54.511661 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:54.511654    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"
Oct 30 20:09:54.512207 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:54.511681    5408 scope.go:122] "RemoveContainer" containerID="7bc1ea7b6f1295d0a79141dd760f9fa92bd3c42bd9892651aec4d7b8b090bc04"
Oct 30 20:09:54.541460 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:54.541436    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"]
Oct 30 20:09:54.547204 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:54.547181    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-19f2ee/test-model-19f2ee-kserve-55549f997f-wwrmw"]
Oct 30 20:09:55.534085 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:55.534055    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154e4c02-a2b9-4dce-aec0-57c62527b2c4" path="/var/lib/kubelet/pods/154e4c02-a2b9-4dce-aec0-57c62527b2c4/volumes"
Oct 30 20:09:56.772108 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:09:56.772068    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"]
Oct 30 20:10:00.716398 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:00.716361    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"]
Oct 30 20:10:00.837337 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:00.837280    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2"]
Oct 30 20:10:00.838508 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:00.838485    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f46abb11-ef94-4122-aab3-d1feaaa46d58" containerName="revoke-keys"
Oct 30 20:10:00.838647 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:00.838636    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46abb11-ef94-4122-aab3-d1feaaa46d58" containerName="revoke-keys"
Oct 30 20:10:00.838721 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:00.838710    5408 cpu_manager.go:414] "containerMap: removing container" podUID="8300c88b-76da-4c18-b214-49d9ae483bd7" containerName="main"
Oct 30 20:10:00.838824 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:00.838815    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8300c88b-76da-4c18-b214-49d9ae483bd7" containerName="main"
Oct 30 20:10:00.838900 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:00.838892    5408 cpu_manager.go:414] "containerMap: removing container" podUID="154e4c02-a2b9-4dce-aec0-57c62527b2c4" containerName="main"
Oct 30 20:10:00.838963 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:00.838955    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154e4c02-a2b9-4dce-aec0-57c62527b2c4" containerName="main"
Oct 30 20:10:00.839254 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:00.839240    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="8300c88b-76da-4c18-b214-49d9ae483bd7" containerName="main"
Oct 30 20:10:00.839366 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:00.839356    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="154e4c02-a2b9-4dce-aec0-57c62527b2c4" containerName="main"
Oct 30 20:10:00.839454 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:00.839447    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46abb11-ef94-4122-aab3-d1feaaa46d58" containerName="revoke-keys"
Oct 30 20:10:00.847785 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:00.847751    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2"
Oct 30 20:10:00.851815 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:00.851792    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2"]
Oct 30 20:10:00.856281 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:00.856262    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:10:00.856641 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:00.856597    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:10:00.978518 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:00.978427    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ff1a51-6102-46df-9a93-cb2144161bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16c3b5db-vkrf2\" (UID: \"f8ff1a51-6102-46df-9a93-cb2144161bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2"
Oct 30 20:10:00.978692 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:00.978681    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l868\" (UniqueName: \"kubernetes.io/projected/f8ff1a51-6102-46df-9a93-cb2144161bf3-kube-api-access-5l868\") pod \"maas-api-revoke-sub-16c3b5db-vkrf2\" (UID: \"f8ff1a51-6102-46df-9a93-cb2144161bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2"
Oct 30 20:10:00.978825 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:00.978797    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ff1a51-6102-46df-9a93-cb2144161bf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16c3b5db-vkrf2\" (UID: \"f8ff1a51-6102-46df-9a93-cb2144161bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2"
Oct 30 20:10:01.080223 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:01.080186    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ff1a51-6102-46df-9a93-cb2144161bf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16c3b5db-vkrf2\" (UID: \"f8ff1a51-6102-46df-9a93-cb2144161bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2"
Oct 30 20:10:01.080408 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:01.080342    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ff1a51-6102-46df-9a93-cb2144161bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16c3b5db-vkrf2\" (UID: \"f8ff1a51-6102-46df-9a93-cb2144161bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2"
Oct 30 20:10:01.080630 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:01.080610    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l868\" (UniqueName: \"kubernetes.io/projected/f8ff1a51-6102-46df-9a93-cb2144161bf3-kube-api-access-5l868\") pod \"maas-api-revoke-sub-16c3b5db-vkrf2\" (UID: \"f8ff1a51-6102-46df-9a93-cb2144161bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2"
Oct 30 20:10:01.081047 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:01.081016    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ff1a51-6102-46df-9a93-cb2144161bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16c3b5db-vkrf2\" (UID: \"f8ff1a51-6102-46df-9a93-cb2144161bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2"
Oct 30 20:10:01.082565 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:01.082545    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ff1a51-6102-46df-9a93-cb2144161bf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16c3b5db-vkrf2\" (UID: \"f8ff1a51-6102-46df-9a93-cb2144161bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2"
Oct 30 20:10:01.121870 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:01.121839    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l868\" (UniqueName: \"kubernetes.io/projected/f8ff1a51-6102-46df-9a93-cb2144161bf3-kube-api-access-5l868\") pod \"maas-api-revoke-sub-16c3b5db-vkrf2\" (UID: \"f8ff1a51-6102-46df-9a93-cb2144161bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2"
Oct 30 20:10:01.168817 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:01.168792    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2"
Oct 30 20:10:01.481088 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:01.481004    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2"]
Oct 30 20:10:01.481513 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:10:01.481487    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ff1a51_6102_46df_9a93_cb2144161bf3.slice/crio-0169dde0a573a8e39e340995bcb696bb0f48d39b6c0d70fae08bd212a134ed6b WatchSource:0}: Error finding container 0169dde0a573a8e39e340995bcb696bb0f48d39b6c0d70fae08bd212a134ed6b: Status 404 returned error can't find the container with id 0169dde0a573a8e39e340995bcb696bb0f48d39b6c0d70fae08bd212a134ed6b
Oct 30 20:10:01.873415 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:01.873379    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2" event={"ID":"f8ff1a51-6102-46df-9a93-cb2144161bf3","Type":"ContainerStarted","Data":"0169dde0a573a8e39e340995bcb696bb0f48d39b6c0d70fae08bd212a134ed6b"}
Oct 30 20:10:02.714927 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:02.714899    5408 ???:1] "http: TLS handshake error from 10.0.139.103:45365: EOF"
Oct 30 20:10:02.883322 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:02.883238    5408 generic.go:356] "Generic (PLEG): container finished" podID="f8ff1a51-6102-46df-9a93-cb2144161bf3" containerID="3f16c1ffdfaecb09ebdd817ae433b9ef043340b16ecc6b54a96ff0fd1fcf0744" exitCode=0
Oct 30 20:10:02.883703 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:02.883317    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2" event={"ID":"f8ff1a51-6102-46df-9a93-cb2144161bf3","Type":"ContainerDied","Data":"3f16c1ffdfaecb09ebdd817ae433b9ef043340b16ecc6b54a96ff0fd1fcf0744"}
Oct 30 20:10:03.942467 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:03.942438    5408 ???:1] "http: TLS handshake error from 10.0.139.103:41133: EOF"
Oct 30 20:10:04.108046 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:04.108002    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2"
Oct 30 20:10:04.197871 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:04.197771    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8ff1a51-6102-46df-9a93-cb2144161bf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ff1a51-6102-46df-9a93-cb2144161bf3-maas-api-xks-ca\") pod \"f8ff1a51-6102-46df-9a93-cb2144161bf3\" (UID: \"f8ff1a51-6102-46df-9a93-cb2144161bf3\") "
Oct 30 20:10:04.198120 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:04.197889    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ff1a51-6102-46df-9a93-cb2144161bf3-kube-api-access-5l868\" (UniqueName: \"kubernetes.io/projected/f8ff1a51-6102-46df-9a93-cb2144161bf3-kube-api-access-5l868\") pod \"f8ff1a51-6102-46df-9a93-cb2144161bf3\" (UID: \"f8ff1a51-6102-46df-9a93-cb2144161bf3\") "
Oct 30 20:10:04.198459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:04.198430    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8ff1a51-6102-46df-9a93-cb2144161bf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ff1a51-6102-46df-9a93-cb2144161bf3-maas-api-openshift-ca\") pod \"f8ff1a51-6102-46df-9a93-cb2144161bf3\" (UID: \"f8ff1a51-6102-46df-9a93-cb2144161bf3\") "
Oct 30 20:10:04.200528 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:04.198851    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ff1a51-6102-46df-9a93-cb2144161bf3-maas-api-openshift-ca" pod "f8ff1a51-6102-46df-9a93-cb2144161bf3" (UID: "f8ff1a51-6102-46df-9a93-cb2144161bf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:04.200528 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:04.199370    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ff1a51-6102-46df-9a93-cb2144161bf3-maas-api-openshift-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:04.201152 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:04.201121    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ff1a51-6102-46df-9a93-cb2144161bf3-kube-api-access-5l868" pod "f8ff1a51-6102-46df-9a93-cb2144161bf3" (UID: "f8ff1a51-6102-46df-9a93-cb2144161bf3"). InnerVolumeSpecName "kube-api-access-5l868". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:04.203738 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:04.203712    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ff1a51-6102-46df-9a93-cb2144161bf3-maas-api-xks-ca" pod "f8ff1a51-6102-46df-9a93-cb2144161bf3" (UID: "f8ff1a51-6102-46df-9a93-cb2144161bf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:04.300914 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:04.300885    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ff1a51-6102-46df-9a93-cb2144161bf3-maas-api-xks-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:04.300914 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:04.300912    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l868\" (UniqueName: \"kubernetes.io/projected/f8ff1a51-6102-46df-9a93-cb2144161bf3-kube-api-access-5l868\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:04.900595 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:04.900558    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2" event={"ID":"f8ff1a51-6102-46df-9a93-cb2144161bf3","Type":"ContainerDied","Data":"0169dde0a573a8e39e340995bcb696bb0f48d39b6c0d70fae08bd212a134ed6b"}
Oct 30 20:10:04.900595 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:04.900592    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2"
Oct 30 20:10:04.900862 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:04.900608    5408 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0169dde0a573a8e39e340995bcb696bb0f48d39b6c0d70fae08bd212a134ed6b"
Oct 30 20:10:09.247764 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:09.247732    5408 ???:1] "http: TLS handshake error from 10.0.139.103:44116: read tcp 10.0.141.19:10250->10.0.139.103:44116: read: connection reset by peer"
Oct 30 20:10:09.745830 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:10:09.743393    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:10:11.797650 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:11.797599    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr" podUID="ea5acb91-6d26-4368-bbf8-a40eaa8d7d24" containerName="main" containerID="cri-o://61bbc667a8c050eddc0783fee135dedaec755c31519ea055cdbeea40d7be7464" gracePeriod=45
Oct 30 20:10:11.963594 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:11.963557    5408 generic.go:356] "Generic (PLEG): container finished" podID="ea5acb91-6d26-4368-bbf8-a40eaa8d7d24" containerID="61bbc667a8c050eddc0783fee135dedaec755c31519ea055cdbeea40d7be7464" exitCode=0
Oct 30 20:10:11.963805 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:11.963786    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr" event={"ID":"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24","Type":"ContainerDied","Data":"61bbc667a8c050eddc0783fee135dedaec755c31519ea055cdbeea40d7be7464"}
Oct 30 20:10:12.147193 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.147165    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:10:12.248806 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.248775    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-tmp-dir\") pod \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") "
Oct 30 20:10:12.249019 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.248874    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-dshm\") pod \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") "
Oct 30 20:10:12.249110 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.249081    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-tmp-dir" pod "ea5acb91-6d26-4368-bbf8-a40eaa8d7d24" (UID: "ea5acb91-6d26-4368-bbf8-a40eaa8d7d24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:12.249110 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.249102    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-tls-certs\") pod \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") "
Oct 30 20:10:12.249225 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.249186    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-home\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-home\") pod \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") "
Oct 30 20:10:12.249408 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.249382    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-model-cache\") pod \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") "
Oct 30 20:10:12.249489 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.249437    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-kube-api-access-6tqx2\" (UniqueName: \"kubernetes.io/projected/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-kube-api-access-6tqx2\") pod \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\" (UID: \"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24\") "
Oct 30 20:10:12.249563 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.249438    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-home" pod "ea5acb91-6d26-4368-bbf8-a40eaa8d7d24" (UID: "ea5acb91-6d26-4368-bbf8-a40eaa8d7d24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:12.250205 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.249672    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-model-cache" pod "ea5acb91-6d26-4368-bbf8-a40eaa8d7d24" (UID: "ea5acb91-6d26-4368-bbf8-a40eaa8d7d24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:12.250711 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.250689    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:12.250800 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.250714    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:12.250800 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.250729    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:12.251860 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.251816    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-kube-api-access-6tqx2" pod "ea5acb91-6d26-4368-bbf8-a40eaa8d7d24" (UID: "ea5acb91-6d26-4368-bbf8-a40eaa8d7d24"). InnerVolumeSpecName "kube-api-access-6tqx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:12.252002 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.251949    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-dshm" pod "ea5acb91-6d26-4368-bbf8-a40eaa8d7d24" (UID: "ea5acb91-6d26-4368-bbf8-a40eaa8d7d24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:12.252111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.252089    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-tls-certs" pod "ea5acb91-6d26-4368-bbf8-a40eaa8d7d24" (UID: "ea5acb91-6d26-4368-bbf8-a40eaa8d7d24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:12.353009 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.352972    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tqx2\" (UniqueName: \"kubernetes.io/projected/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-kube-api-access-6tqx2\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:12.353009 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.353008    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:12.353282 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.353022    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:12.988923 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.988891    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"
Oct 30 20:10:12.988923 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.988907    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr" event={"ID":"ea5acb91-6d26-4368-bbf8-a40eaa8d7d24","Type":"ContainerDied","Data":"cd00dc88d1d5aae9270bf2dd86d33ac2318d6a2e94ec43f330383a51bed927a5"}
Oct 30 20:10:12.989578 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:12.988956    5408 scope.go:122] "RemoveContainer" containerID="61bbc667a8c050eddc0783fee135dedaec755c31519ea055cdbeea40d7be7464"
Oct 30 20:10:13.023120 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:13.023088    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"]
Oct 30 20:10:13.027737 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:13.027716    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1f5eee/test-model-1f5eee-kserve-7c46b74789-mr4vr"]
Oct 30 20:10:13.539200 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:13.539163    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5acb91-6d26-4368-bbf8-a40eaa8d7d24" path="/var/lib/kubelet/pods/ea5acb91-6d26-4368-bbf8-a40eaa8d7d24/volumes"
Oct 30 20:10:15.785448 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:15.785339    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn" podUID="0dd5e806-31e7-4ae8-90d3-7806662cfd16" containerName="main" containerID="cri-o://87612665866e9be3f6d6967c4bd812adb99570aa39dbf99d720a36e2cbff4fb9" gracePeriod=45
Oct 30 20:10:15.959367 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:15.959331    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"]
Oct 30 20:10:16.343649 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.343603    5408 generic.go:356] "Generic (PLEG): container finished" podID="0dd5e806-31e7-4ae8-90d3-7806662cfd16" containerID="87612665866e9be3f6d6967c4bd812adb99570aa39dbf99d720a36e2cbff4fb9" exitCode=0
Oct 30 20:10:16.352178 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.343871    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn" event={"ID":"0dd5e806-31e7-4ae8-90d3-7806662cfd16","Type":"ContainerDied","Data":"87612665866e9be3f6d6967c4bd812adb99570aa39dbf99d720a36e2cbff4fb9"}
Oct 30 20:10:16.436513 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.436485    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:10:16.479323 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.479285    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4"]
Oct 30 20:10:16.480739 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.480686    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f8ff1a51-6102-46df-9a93-cb2144161bf3" containerName="revoke-keys"
Oct 30 20:10:16.480739 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.480714    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ff1a51-6102-46df-9a93-cb2144161bf3" containerName="revoke-keys"
Oct 30 20:10:16.480924 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.480754    5408 cpu_manager.go:414] "containerMap: removing container" podUID="ea5acb91-6d26-4368-bbf8-a40eaa8d7d24" containerName="main"
Oct 30 20:10:16.480924 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.480763    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5acb91-6d26-4368-bbf8-a40eaa8d7d24" containerName="main"
Oct 30 20:10:16.480924 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.480797    5408 cpu_manager.go:414] "containerMap: removing container" podUID="0dd5e806-31e7-4ae8-90d3-7806662cfd16" containerName="main"
Oct 30 20:10:16.480924 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.480802    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd5e806-31e7-4ae8-90d3-7806662cfd16" containerName="main"
Oct 30 20:10:16.481169 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.481000    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ff1a51-6102-46df-9a93-cb2144161bf3" containerName="revoke-keys"
Oct 30 20:10:16.481169 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.481015    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5acb91-6d26-4368-bbf8-a40eaa8d7d24" containerName="main"
Oct 30 20:10:16.481169 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.481046    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd5e806-31e7-4ae8-90d3-7806662cfd16" containerName="main"
Oct 30 20:10:16.491483 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.491414    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4"
Oct 30 20:10:16.508573 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.508552    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:10:16.543888 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.533876    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd5e806-31e7-4ae8-90d3-7806662cfd16-kube-api-access-4kv6g\" (UniqueName: \"kubernetes.io/projected/0dd5e806-31e7-4ae8-90d3-7806662cfd16-kube-api-access-4kv6g\") pod \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") "
Oct 30 20:10:16.544286 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.544262    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dd5e806-31e7-4ae8-90d3-7806662cfd16-tls-certs\" (UniqueName: \"kubernetes.io/secret/0dd5e806-31e7-4ae8-90d3-7806662cfd16-tls-certs\") pod \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") "
Oct 30 20:10:16.544595 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.544575    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-tmp-dir\") pod \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") "
Oct 30 20:10:16.544752 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.544737    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-home\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-home\") pod \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") "
Oct 30 20:10:16.544865 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.544853    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-dshm\") pod \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") "
Oct 30 20:10:16.548332 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.545100    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-model-cache\") pod \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\" (UID: \"0dd5e806-31e7-4ae8-90d3-7806662cfd16\") "
Oct 30 20:10:16.548900 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.548873    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4\" (UID: \"07a25adf-6a0b-44fe-a31b-b85b1d44fc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4"
Oct 30 20:10:16.549466 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.549422    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4w4l\" (UniqueName: \"kubernetes.io/projected/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-kube-api-access-l4w4l\") pod \"maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4\" (UID: \"07a25adf-6a0b-44fe-a31b-b85b1d44fc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4"
Oct 30 20:10:16.549830 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.549783    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4\" (UID: \"07a25adf-6a0b-44fe-a31b-b85b1d44fc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4"
Oct 30 20:10:16.550487 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.550262    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-tmp-dir" pod "0dd5e806-31e7-4ae8-90d3-7806662cfd16" (UID: "0dd5e806-31e7-4ae8-90d3-7806662cfd16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:16.550487 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.534841    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4"]
Oct 30 20:10:16.550487 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.550448    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-home" pod "0dd5e806-31e7-4ae8-90d3-7806662cfd16" (UID: "0dd5e806-31e7-4ae8-90d3-7806662cfd16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:16.554226 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.554126    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-dshm" pod "0dd5e806-31e7-4ae8-90d3-7806662cfd16" (UID: "0dd5e806-31e7-4ae8-90d3-7806662cfd16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:16.561229 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.559293    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"]
Oct 30 20:10:16.561229 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.559529    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7" podUID="4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" containerName="istio-proxy" containerID="cri-o://3ce0d393151d17b2bf171fd0af4188458c3eb1b356a174ac9db0248abb57e381" gracePeriod=30
Oct 30 20:10:16.564013 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.563966    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd5e806-31e7-4ae8-90d3-7806662cfd16-tls-certs" pod "0dd5e806-31e7-4ae8-90d3-7806662cfd16" (UID: "0dd5e806-31e7-4ae8-90d3-7806662cfd16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:16.569055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.567379    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"]
Oct 30 20:10:16.569055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.567626    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerName="istio-proxy" containerID="cri-o://d80ce52ae2d7be751312ece6e8ef83d805957bb8d83aa34af42f60928ac70501" gracePeriod=30
Oct 30 20:10:16.570157 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.569847    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd5e806-31e7-4ae8-90d3-7806662cfd16-kube-api-access-4kv6g" pod "0dd5e806-31e7-4ae8-90d3-7806662cfd16" (UID: "0dd5e806-31e7-4ae8-90d3-7806662cfd16"). InnerVolumeSpecName "kube-api-access-4kv6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:16.573169 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.573137    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-model-cache" pod "0dd5e806-31e7-4ae8-90d3-7806662cfd16" (UID: "0dd5e806-31e7-4ae8-90d3-7806662cfd16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:16.598156 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.598047    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:10:16.687307 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.666092    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4\" (UID: \"07a25adf-6a0b-44fe-a31b-b85b1d44fc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4"
Oct 30 20:10:16.687307 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.666498    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4w4l\" (UniqueName: \"kubernetes.io/projected/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-kube-api-access-l4w4l\") pod \"maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4\" (UID: \"07a25adf-6a0b-44fe-a31b-b85b1d44fc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4"
Oct 30 20:10:16.687307 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.666666    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4\" (UID: \"07a25adf-6a0b-44fe-a31b-b85b1d44fc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4"
Oct 30 20:10:16.687307 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.666758    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dd5e806-31e7-4ae8-90d3-7806662cfd16-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:16.687307 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.666777    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:16.687307 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.666790    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:16.687307 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.666803    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:16.687307 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.666816    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dd5e806-31e7-4ae8-90d3-7806662cfd16-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:16.687307 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.666830    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kv6g\" (UniqueName: \"kubernetes.io/projected/0dd5e806-31e7-4ae8-90d3-7806662cfd16-kube-api-access-4kv6g\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:16.687307 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.671948    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4\" (UID: \"07a25adf-6a0b-44fe-a31b-b85b1d44fc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4"
Oct 30 20:10:16.699787 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.699748    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4\" (UID: \"07a25adf-6a0b-44fe-a31b-b85b1d44fc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4"
Oct 30 20:10:16.703093 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.702478    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4w4l\" (UniqueName: \"kubernetes.io/projected/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-kube-api-access-l4w4l\") pod \"maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4\" (UID: \"07a25adf-6a0b-44fe-a31b-b85b1d44fc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4"
Oct 30 20:10:16.822666 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.822630    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4"
Oct 30 20:10:16.913426 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.913331    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"]
Oct 30 20:10:16.919916 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:16.914021    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2" podUID="795803a2-94c7-4773-b51c-5f84a33a28f7" containerName="istio-proxy" containerID="cri-o://20f09078c860f497fb00f4f3a22f1a58a3137914a04180b14b7641061e9752a7" gracePeriod=30
Oct 30 20:10:17.193273 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:17.189947    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4"]
Oct 30 20:10:17.356474 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:17.356441    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"
Oct 30 20:10:17.356622 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:17.356472    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn" event={"ID":"0dd5e806-31e7-4ae8-90d3-7806662cfd16","Type":"ContainerDied","Data":"53c53eff776f41c4b12adfb91472dc95a9b156d5e85114cf6fae8a832fe3b542"}
Oct 30 20:10:17.356622 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:17.356522    5408 scope.go:122] "RemoveContainer" containerID="87612665866e9be3f6d6967c4bd812adb99570aa39dbf99d720a36e2cbff4fb9"
Oct 30 20:10:17.358374 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:17.358348    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4" event={"ID":"07a25adf-6a0b-44fe-a31b-b85b1d44fc7d","Type":"ContainerStarted","Data":"d1419d87c23e5ca378b46bf601f652c278fd1dae5eafd5f23ddea9e9a293c77b"}
Oct 30 20:10:17.390625 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:17.390590    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"]
Oct 30 20:10:17.403988 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:17.403953    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7f7d21/test-model-7f7d21-kserve-594d95cbf7-l2ntn"]
Oct 30 20:10:17.611977 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:17.611948    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd5e806-31e7-4ae8-90d3-7806662cfd16" path="/var/lib/kubelet/pods/0dd5e806-31e7-4ae8-90d3-7806662cfd16/volumes"
Oct 30 20:10:18.373179 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:18.373149    5408 generic.go:356] "Generic (PLEG): container finished" podID="07a25adf-6a0b-44fe-a31b-b85b1d44fc7d" containerID="8ab2a93c1de41d19a4bf688736f37a059678061b040e21966340f9aad552e169" exitCode=0
Oct 30 20:10:18.373591 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:18.373203    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4" event={"ID":"07a25adf-6a0b-44fe-a31b-b85b1d44fc7d","Type":"ContainerDied","Data":"8ab2a93c1de41d19a4bf688736f37a059678061b040e21966340f9aad552e169"}
Oct 30 20:10:19.602539 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:19.602513    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4"
Oct 30 20:10:19.744350 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:19.744317    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-kube-api-access-l4w4l\" (UniqueName: \"kubernetes.io/projected/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-kube-api-access-l4w4l\") pod \"07a25adf-6a0b-44fe-a31b-b85b1d44fc7d\" (UID: \"07a25adf-6a0b-44fe-a31b-b85b1d44fc7d\") "
Oct 30 20:10:19.744547 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:19.744498    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-maas-api-openshift-ca\") pod \"07a25adf-6a0b-44fe-a31b-b85b1d44fc7d\" (UID: \"07a25adf-6a0b-44fe-a31b-b85b1d44fc7d\") "
Oct 30 20:10:19.744854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:19.744670    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-maas-api-xks-ca\") pod \"07a25adf-6a0b-44fe-a31b-b85b1d44fc7d\" (UID: \"07a25adf-6a0b-44fe-a31b-b85b1d44fc7d\") "
Oct 30 20:10:19.745071 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:19.745023    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-maas-api-openshift-ca" pod "07a25adf-6a0b-44fe-a31b-b85b1d44fc7d" (UID: "07a25adf-6a0b-44fe-a31b-b85b1d44fc7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:19.745466 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:19.745442    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-maas-api-openshift-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:19.746785 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:19.746761    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-kube-api-access-l4w4l" pod "07a25adf-6a0b-44fe-a31b-b85b1d44fc7d" (UID: "07a25adf-6a0b-44fe-a31b-b85b1d44fc7d"). InnerVolumeSpecName "kube-api-access-l4w4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:19.747175 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:19.747149    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-maas-api-xks-ca" pod "07a25adf-6a0b-44fe-a31b-b85b1d44fc7d" (UID: "07a25adf-6a0b-44fe-a31b-b85b1d44fc7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:19.849427 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:19.847017    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-maas-api-xks-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:19.849427 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:19.847106    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4w4l\" (UniqueName: \"kubernetes.io/projected/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d-kube-api-access-l4w4l\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:20.092726 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:20.092651    5408 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:10:20.092726 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:20.092710    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7" podUID="4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:10:20.395455 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:20.395381    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4"
Oct 30 20:10:20.395844 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:20.395496    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4" event={"ID":"07a25adf-6a0b-44fe-a31b-b85b1d44fc7d","Type":"ContainerDied","Data":"d1419d87c23e5ca378b46bf601f652c278fd1dae5eafd5f23ddea9e9a293c77b"}
Oct 30 20:10:20.395844 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:20.395540    5408 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1419d87c23e5ca378b46bf601f652c278fd1dae5eafd5f23ddea9e9a293c77b"
Oct 30 20:10:21.164237 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:21.160232    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"]
Oct 30 20:10:22.092780 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.092754    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 20:10:22.174852 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.174818    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-workload-certs\") pod \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") "
Oct 30 20:10:22.175346 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.174870    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-kube-api-access-xhvmx\" (UniqueName: \"kubernetes.io/projected/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-kube-api-access-xhvmx\") pod \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") "
Oct 30 20:10:22.175346 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.174918    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-data\") pod \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") "
Oct 30 20:10:22.175346 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.174962    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-credential-socket\") pod \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") "
Oct 30 20:10:22.175346 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.175014    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istiod-ca-cert\") pod \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") "
Oct 30 20:10:22.175346 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.175055    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-envoy\") pod \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") "
Oct 30 20:10:22.175346 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.175124    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-workload-socket\") pod \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") "
Oct 30 20:10:22.175346 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.175192    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-token\" (UniqueName: \"kubernetes.io/projected/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-token\") pod \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") "
Oct 30 20:10:22.175713 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.175564    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-credential-socket" pod "4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" (UID: "4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.175766 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.175741    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-workload-certs" pod "4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" (UID: "4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.176885 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.176850    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-data" pod "4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" (UID: "4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.177363 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.177337    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istiod-ca-cert" pod "4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" (UID: "4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:22.177532 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.177509    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-workload-socket" pod "4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" (UID: "4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.177619 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.177603    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-podinfo\") pod \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\" (UID: \"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe\") "
Oct 30 20:10:22.178603 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.178576    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.178707 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.178608    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.178707 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.178625    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.178707 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.178639    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.178707 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.178654    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.184219 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.184189    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-podinfo" pod "4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" (UID: "4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:10:22.184312 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.184229    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-token" pod "4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" (UID: "4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:22.188324 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.188286    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-envoy" pod "4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" (UID: "4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.188842 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.188809    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-kube-api-access-xhvmx" pod "4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" (UID: "4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe"). InnerVolumeSpecName "kube-api-access-xhvmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:22.280322 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.280229    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.280322 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.280264    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhvmx\" (UniqueName: \"kubernetes.io/projected/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-kube-api-access-xhvmx\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.280322 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.280282    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.280322 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.280297    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.281655 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.280984    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 20:10:22.354405 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.354377    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 20:10:22.381261 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.381235    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a746e113-69a0-4d13-acf1-bb28ba1769b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a746e113-69a0-4d13-acf1-bb28ba1769b7-istiod-ca-cert\") pod \"a746e113-69a0-4d13-acf1-bb28ba1769b7\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") "
Oct 30 20:10:22.381501 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.381293    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-envoy\") pod \"a746e113-69a0-4d13-acf1-bb28ba1769b7\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") "
Oct 30 20:10:22.381501 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.381357    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-envoy\") pod \"795803a2-94c7-4773-b51c-5f84a33a28f7\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") "
Oct 30 20:10:22.381501 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.381381    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-workload-socket\") pod \"a746e113-69a0-4d13-acf1-bb28ba1769b7\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") "
Oct 30 20:10:22.381501 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.381418    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-workload-socket\") pod \"795803a2-94c7-4773-b51c-5f84a33a28f7\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") "
Oct 30 20:10:22.381501 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.381451    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-workload-certs\") pod \"795803a2-94c7-4773-b51c-5f84a33a28f7\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") "
Oct 30 20:10:22.381725 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.381520    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-credential-socket\") pod \"795803a2-94c7-4773-b51c-5f84a33a28f7\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") "
Oct 30 20:10:22.381725 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.381590    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-data\") pod \"795803a2-94c7-4773-b51c-5f84a33a28f7\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") "
Oct 30 20:10:22.381725 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.381661    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a746e113-69a0-4d13-acf1-bb28ba1769b7-istiod-ca-cert" pod "a746e113-69a0-4d13-acf1-bb28ba1769b7" (UID: "a746e113-69a0-4d13-acf1-bb28ba1769b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:22.381991 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.381968    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-workload-socket" pod "795803a2-94c7-4773-b51c-5f84a33a28f7" (UID: "795803a2-94c7-4773-b51c-5f84a33a28f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.382470 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.382421    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-credential-socket" pod "a746e113-69a0-4d13-acf1-bb28ba1769b7" (UID: "a746e113-69a0-4d13-acf1-bb28ba1769b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.382631 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.382491    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-workload-socket" pod "a746e113-69a0-4d13-acf1-bb28ba1769b7" (UID: "a746e113-69a0-4d13-acf1-bb28ba1769b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.382718 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.382676    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-credential-socket\") pod \"a746e113-69a0-4d13-acf1-bb28ba1769b7\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") "
Oct 30 20:10:22.382794 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.382725    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-credential-socket" pod "795803a2-94c7-4773-b51c-5f84a33a28f7" (UID: "795803a2-94c7-4773-b51c-5f84a33a28f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.382794 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.382738    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-data\") pod \"a746e113-69a0-4d13-acf1-bb28ba1769b7\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") "
Oct 30 20:10:22.382794 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.382773    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a746e113-69a0-4d13-acf1-bb28ba1769b7-kube-api-access-4f9ks\" (UniqueName: \"kubernetes.io/projected/a746e113-69a0-4d13-acf1-bb28ba1769b7-kube-api-access-4f9ks\") pod \"a746e113-69a0-4d13-acf1-bb28ba1769b7\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") "
Oct 30 20:10:22.382960 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.382920    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-workload-certs" pod "795803a2-94c7-4773-b51c-5f84a33a28f7" (UID: "795803a2-94c7-4773-b51c-5f84a33a28f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.382960 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.382935    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-workload-certs\") pod \"a746e113-69a0-4d13-acf1-bb28ba1769b7\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") "
Oct 30 20:10:22.383080 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.383015    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-token\" (UniqueName: \"kubernetes.io/projected/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-token\") pod \"795803a2-94c7-4773-b51c-5f84a33a28f7\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") "
Oct 30 20:10:22.383127 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.383113    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-workload-certs" pod "a746e113-69a0-4d13-acf1-bb28ba1769b7" (UID: "a746e113-69a0-4d13-acf1-bb28ba1769b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.383380 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.383353    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-data" pod "a746e113-69a0-4d13-acf1-bb28ba1769b7" (UID: "a746e113-69a0-4d13-acf1-bb28ba1769b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.383514 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.383427    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-podinfo\") pod \"795803a2-94c7-4773-b51c-5f84a33a28f7\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") "
Oct 30 20:10:22.383583 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.383515    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-token\" (UniqueName: \"kubernetes.io/projected/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-token\") pod \"a746e113-69a0-4d13-acf1-bb28ba1769b7\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") "
Oct 30 20:10:22.383640 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.383586    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/795803a2-94c7-4773-b51c-5f84a33a28f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/795803a2-94c7-4773-b51c-5f84a33a28f7-istiod-ca-cert\") pod \"795803a2-94c7-4773-b51c-5f84a33a28f7\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") "
Oct 30 20:10:22.383693 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.383644    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795803a2-94c7-4773-b51c-5f84a33a28f7-kube-api-access-5qmhx\" (UniqueName: \"kubernetes.io/projected/795803a2-94c7-4773-b51c-5f84a33a28f7-kube-api-access-5qmhx\") pod \"795803a2-94c7-4773-b51c-5f84a33a28f7\" (UID: \"795803a2-94c7-4773-b51c-5f84a33a28f7\") "
Oct 30 20:10:22.383693 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.383675    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-podinfo\") pod \"a746e113-69a0-4d13-acf1-bb28ba1769b7\" (UID: \"a746e113-69a0-4d13-acf1-bb28ba1769b7\") "
Oct 30 20:10:22.384583 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.384551    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795803a2-94c7-4773-b51c-5f84a33a28f7-istiod-ca-cert" pod "795803a2-94c7-4773-b51c-5f84a33a28f7" (UID: "795803a2-94c7-4773-b51c-5f84a33a28f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:22.385017 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.384993    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.385162 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.385022    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.385162 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.385085    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.385162 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.385100    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.385162 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.385116    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.385162 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.385129    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.385162 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.385143    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.385162 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.385155    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/795803a2-94c7-4773-b51c-5f84a33a28f7-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.385424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.385171    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a746e113-69a0-4d13-acf1-bb28ba1769b7-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.385583 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.385531    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-token" pod "795803a2-94c7-4773-b51c-5f84a33a28f7" (UID: "795803a2-94c7-4773-b51c-5f84a33a28f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:22.385723 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.385653    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-data" pod "795803a2-94c7-4773-b51c-5f84a33a28f7" (UID: "795803a2-94c7-4773-b51c-5f84a33a28f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.386513 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.386480    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-envoy" pod "a746e113-69a0-4d13-acf1-bb28ba1769b7" (UID: "a746e113-69a0-4d13-acf1-bb28ba1769b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.386628 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.386533    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a746e113-69a0-4d13-acf1-bb28ba1769b7-kube-api-access-4f9ks" pod "a746e113-69a0-4d13-acf1-bb28ba1769b7" (UID: "a746e113-69a0-4d13-acf1-bb28ba1769b7"). InnerVolumeSpecName "kube-api-access-4f9ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:22.386628 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.386558    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-podinfo" pod "795803a2-94c7-4773-b51c-5f84a33a28f7" (UID: "795803a2-94c7-4773-b51c-5f84a33a28f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:10:22.386755 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.386726    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795803a2-94c7-4773-b51c-5f84a33a28f7-kube-api-access-5qmhx" pod "795803a2-94c7-4773-b51c-5f84a33a28f7" (UID: "795803a2-94c7-4773-b51c-5f84a33a28f7"). InnerVolumeSpecName "kube-api-access-5qmhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:22.387307 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.387283    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-podinfo" pod "a746e113-69a0-4d13-acf1-bb28ba1769b7" (UID: "a746e113-69a0-4d13-acf1-bb28ba1769b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:10:22.387498 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.387476    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-token" pod "a746e113-69a0-4d13-acf1-bb28ba1769b7" (UID: "a746e113-69a0-4d13-acf1-bb28ba1769b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:22.388290 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.388268    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-envoy" pod "795803a2-94c7-4773-b51c-5f84a33a28f7" (UID: "795803a2-94c7-4773-b51c-5f84a33a28f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.486566 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.486528    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.486566 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.486558    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.486566 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.486571    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f9ks\" (UniqueName: \"kubernetes.io/projected/a746e113-69a0-4d13-acf1-bb28ba1769b7-kube-api-access-4f9ks\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.486834 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.486583    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.486834 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.486592    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/795803a2-94c7-4773-b51c-5f84a33a28f7-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.486834 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.486602    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.486834 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.486611    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qmhx\" (UniqueName: \"kubernetes.io/projected/795803a2-94c7-4773-b51c-5f84a33a28f7-kube-api-access-5qmhx\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.486834 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.486620    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.486834 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.486629    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a746e113-69a0-4d13-acf1-bb28ba1769b7-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.763047 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.762995    5408 generic.go:356] "Generic (PLEG): container finished" podID="795803a2-94c7-4773-b51c-5f84a33a28f7" containerID="20f09078c860f497fb00f4f3a22f1a58a3137914a04180b14b7641061e9752a7" exitCode=0
Oct 30 20:10:22.765347 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.763385    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"
Oct 30 20:10:22.781645 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.763406    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2" event={"ID":"795803a2-94c7-4773-b51c-5f84a33a28f7","Type":"ContainerDied","Data":"20f09078c860f497fb00f4f3a22f1a58a3137914a04180b14b7641061e9752a7"}
Oct 30 20:10:22.781645 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.765576    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2" event={"ID":"795803a2-94c7-4773-b51c-5f84a33a28f7","Type":"ContainerDied","Data":"a58913f59822dfbfce038e59b2d89beae38698f5cca7951d43570c9d6c147d3a"}
Oct 30 20:10:22.781645 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.765614    5408 scope.go:122] "RemoveContainer" containerID="20f09078c860f497fb00f4f3a22f1a58a3137914a04180b14b7641061e9752a7"
Oct 30 20:10:22.797299 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.797271    5408 generic.go:356] "Generic (PLEG): container finished" podID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerID="d80ce52ae2d7be751312ece6e8ef83d805957bb8d83aa34af42f60928ac70501" exitCode=0
Oct 30 20:10:22.797530 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.797508    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" event={"ID":"a746e113-69a0-4d13-acf1-bb28ba1769b7","Type":"ContainerDied","Data":"d80ce52ae2d7be751312ece6e8ef83d805957bb8d83aa34af42f60928ac70501"}
Oct 30 20:10:22.797637 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.797554    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft" event={"ID":"a746e113-69a0-4d13-acf1-bb28ba1769b7","Type":"ContainerDied","Data":"3252c229407e402bed01e0a9ef043ec81e26bf01c37e0f1dc4ca1648c6ccc225"}
Oct 30 20:10:22.797637 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.797633    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"
Oct 30 20:10:22.801838 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.799249    5408 scope.go:122] "RemoveContainer" containerID="20f09078c860f497fb00f4f3a22f1a58a3137914a04180b14b7641061e9752a7"
Oct 30 20:10:22.801838 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:10:22.799638    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20f09078c860f497fb00f4f3a22f1a58a3137914a04180b14b7641061e9752a7\": container with ID starting with 20f09078c860f497fb00f4f3a22f1a58a3137914a04180b14b7641061e9752a7 not found: ID does not exist" containerID="20f09078c860f497fb00f4f3a22f1a58a3137914a04180b14b7641061e9752a7"
Oct 30 20:10:22.801838 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.799674    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f09078c860f497fb00f4f3a22f1a58a3137914a04180b14b7641061e9752a7"} err="failed to get container status \"20f09078c860f497fb00f4f3a22f1a58a3137914a04180b14b7641061e9752a7\": rpc error: code = NotFound desc = could not find container \"20f09078c860f497fb00f4f3a22f1a58a3137914a04180b14b7641061e9752a7\": container with ID starting with 20f09078c860f497fb00f4f3a22f1a58a3137914a04180b14b7641061e9752a7 not found: ID does not exist"
Oct 30 20:10:22.801838 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.799698    5408 scope.go:122] "RemoveContainer" containerID="d80ce52ae2d7be751312ece6e8ef83d805957bb8d83aa34af42f60928ac70501"
Oct 30 20:10:22.804690 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.803219    5408 generic.go:356] "Generic (PLEG): container finished" podID="4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" containerID="3ce0d393151d17b2bf171fd0af4188458c3eb1b356a174ac9db0248abb57e381" exitCode=0
Oct 30 20:10:22.804690 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.803295    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7" event={"ID":"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe","Type":"ContainerDied","Data":"3ce0d393151d17b2bf171fd0af4188458c3eb1b356a174ac9db0248abb57e381"}
Oct 30 20:10:22.804690 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.803335    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7" event={"ID":"4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe","Type":"ContainerDied","Data":"3af2c3780707ee48c5e82880904a4e71001cf20105664b33d8fd374cf19745fd"}
Oct 30 20:10:22.804690 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.803536    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"
Oct 30 20:10:22.817320 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.817285    5408 scope.go:122] "RemoveContainer" containerID="d80ce52ae2d7be751312ece6e8ef83d805957bb8d83aa34af42f60928ac70501"
Oct 30 20:10:22.817586 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:10:22.817566    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80ce52ae2d7be751312ece6e8ef83d805957bb8d83aa34af42f60928ac70501\": container with ID starting with d80ce52ae2d7be751312ece6e8ef83d805957bb8d83aa34af42f60928ac70501 not found: ID does not exist" containerID="d80ce52ae2d7be751312ece6e8ef83d805957bb8d83aa34af42f60928ac70501"
Oct 30 20:10:22.817652 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.817593    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80ce52ae2d7be751312ece6e8ef83d805957bb8d83aa34af42f60928ac70501"} err="failed to get container status \"d80ce52ae2d7be751312ece6e8ef83d805957bb8d83aa34af42f60928ac70501\": rpc error: code = NotFound desc = could not find container \"d80ce52ae2d7be751312ece6e8ef83d805957bb8d83aa34af42f60928ac70501\": container with ID starting with d80ce52ae2d7be751312ece6e8ef83d805957bb8d83aa34af42f60928ac70501 not found: ID does not exist"
Oct 30 20:10:22.817652 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.817636    5408 scope.go:122] "RemoveContainer" containerID="3ce0d393151d17b2bf171fd0af4188458c3eb1b356a174ac9db0248abb57e381"
Oct 30 20:10:22.828983 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.828960    5408 scope.go:122] "RemoveContainer" containerID="3ce0d393151d17b2bf171fd0af4188458c3eb1b356a174ac9db0248abb57e381"
Oct 30 20:10:22.829336 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:10:22.829288    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce0d393151d17b2bf171fd0af4188458c3eb1b356a174ac9db0248abb57e381\": container with ID starting with 3ce0d393151d17b2bf171fd0af4188458c3eb1b356a174ac9db0248abb57e381 not found: ID does not exist" containerID="3ce0d393151d17b2bf171fd0af4188458c3eb1b356a174ac9db0248abb57e381"
Oct 30 20:10:22.829451 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.829327    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce0d393151d17b2bf171fd0af4188458c3eb1b356a174ac9db0248abb57e381"} err="failed to get container status \"3ce0d393151d17b2bf171fd0af4188458c3eb1b356a174ac9db0248abb57e381\": rpc error: code = NotFound desc = could not find container \"3ce0d393151d17b2bf171fd0af4188458c3eb1b356a174ac9db0248abb57e381\": container with ID starting with 3ce0d393151d17b2bf171fd0af4188458c3eb1b356a174ac9db0248abb57e381 not found: ID does not exist"
Oct 30 20:10:22.830371 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.830351    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"]
Oct 30 20:10:22.835251 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.835231    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dlbx2"]
Oct 30 20:10:22.852386 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.852335    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"]
Oct 30 20:10:22.859857 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.859837    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zbqb7"]
Oct 30 20:10:22.877064 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.877043    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"]
Oct 30 20:10:22.884089 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:22.884069    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-p5sft"]
Oct 30 20:10:23.536937 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:23.536902    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" path="/var/lib/kubelet/pods/4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe/volumes"
Oct 30 20:10:23.537592 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:23.537576    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="795803a2-94c7-4773-b51c-5f84a33a28f7" path="/var/lib/kubelet/pods/795803a2-94c7-4773-b51c-5f84a33a28f7/volumes"
Oct 30 20:10:23.538201 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:23.538177    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" path="/var/lib/kubelet/pods/a746e113-69a0-4d13-acf1-bb28ba1769b7/volumes"
Oct 30 20:10:24.434058 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:24.427157    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x"]
Oct 30 20:10:24.443782 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:24.438414    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-925e04-2697x"]
Oct 30 20:10:25.537243 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:25.537198    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f46abb11-ef94-4122-aab3-d1feaaa46d58" path="/var/lib/kubelet/pods/f46abb11-ef94-4122-aab3-d1feaaa46d58/volumes"
Oct 30 20:10:26.582474 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:26.582433    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg"]
Oct 30 20:10:26.916155 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:26.916060    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"]
Oct 30 20:10:31.370515 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.370473    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz" podUID="cc5cea2d-0de3-445f-923c-44341da06bc8" containerName="main" containerID="cri-o://b08475cfcb3149f00d87dcd7a52dc6df71193984c118ed753332b51f80be1b21" gracePeriod=45
Oct 30 20:10:31.583252 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.583212    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg" podUID="2982e319-9261-44c2-bc78-54bd30e83dea" containerName="payload-pre-processing" containerID="cri-o://29bb6b68d12dab06f4b98d3a8ed4f1dd405a07d8f6afd96204bbf5bd314e89ec" gracePeriod=25
Oct 30 20:10:31.728289 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.728238    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:10:31.740401 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.740366    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-model-cache\") pod \"cc5cea2d-0de3-445f-923c-44341da06bc8\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") "
Oct 30 20:10:31.740602 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.740443    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-dshm\") pod \"cc5cea2d-0de3-445f-923c-44341da06bc8\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") "
Oct 30 20:10:31.740602 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.740500    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc5cea2d-0de3-445f-923c-44341da06bc8-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc5cea2d-0de3-445f-923c-44341da06bc8-tls-certs\") pod \"cc5cea2d-0de3-445f-923c-44341da06bc8\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") "
Oct 30 20:10:31.740774 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.740654    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc5cea2d-0de3-445f-923c-44341da06bc8-kube-api-access-qklct\" (UniqueName: \"kubernetes.io/projected/cc5cea2d-0de3-445f-923c-44341da06bc8-kube-api-access-qklct\") pod \"cc5cea2d-0de3-445f-923c-44341da06bc8\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") "
Oct 30 20:10:31.740774 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.740692    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-model-cache" pod "cc5cea2d-0de3-445f-923c-44341da06bc8" (UID: "cc5cea2d-0de3-445f-923c-44341da06bc8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:31.740774 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.740707    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-home\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-home\") pod \"cc5cea2d-0de3-445f-923c-44341da06bc8\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") "
Oct 30 20:10:31.740774 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.740747    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-tmp-dir\") pod \"cc5cea2d-0de3-445f-923c-44341da06bc8\" (UID: \"cc5cea2d-0de3-445f-923c-44341da06bc8\") "
Oct 30 20:10:31.741460 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.741212    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-home" pod "cc5cea2d-0de3-445f-923c-44341da06bc8" (UID: "cc5cea2d-0de3-445f-923c-44341da06bc8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:31.741460 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.741435    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-tmp-dir" pod "cc5cea2d-0de3-445f-923c-44341da06bc8" (UID: "cc5cea2d-0de3-445f-923c-44341da06bc8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:31.741745 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.741729    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:31.741745 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.741747    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:31.741885 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.741756    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:31.743573 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.743524    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc5cea2d-0de3-445f-923c-44341da06bc8-tls-certs" pod "cc5cea2d-0de3-445f-923c-44341da06bc8" (UID: "cc5cea2d-0de3-445f-923c-44341da06bc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:31.743714 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.743697    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5cea2d-0de3-445f-923c-44341da06bc8-kube-api-access-qklct" pod "cc5cea2d-0de3-445f-923c-44341da06bc8" (UID: "cc5cea2d-0de3-445f-923c-44341da06bc8"). InnerVolumeSpecName "kube-api-access-qklct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:31.744288 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.744256    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-dshm" pod "cc5cea2d-0de3-445f-923c-44341da06bc8" (UID: "cc5cea2d-0de3-445f-923c-44341da06bc8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:31.843856 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.843811    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc5cea2d-0de3-445f-923c-44341da06bc8-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:31.843856 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.843851    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc5cea2d-0de3-445f-923c-44341da06bc8-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:31.844201 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.843868    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qklct\" (UniqueName: \"kubernetes.io/projected/cc5cea2d-0de3-445f-923c-44341da06bc8-kube-api-access-qklct\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:31.883598 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.883577    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg"
Oct 30 20:10:31.944552 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.944453    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2982e319-9261-44c2-bc78-54bd30e83dea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2982e319-9261-44c2-bc78-54bd30e83dea-plugins-config-volume\") pod \"2982e319-9261-44c2-bc78-54bd30e83dea\" (UID: \"2982e319-9261-44c2-bc78-54bd30e83dea\") "
Oct 30 20:10:31.944879 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.944829    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2982e319-9261-44c2-bc78-54bd30e83dea-plugins-config-volume" pod "2982e319-9261-44c2-bc78-54bd30e83dea" (UID: "2982e319-9261-44c2-bc78-54bd30e83dea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:31.944879 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.944873    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2982e319-9261-44c2-bc78-54bd30e83dea-kube-api-access-2cfj4\" (UniqueName: \"kubernetes.io/projected/2982e319-9261-44c2-bc78-54bd30e83dea-kube-api-access-2cfj4\") pod \"2982e319-9261-44c2-bc78-54bd30e83dea\" (UID: \"2982e319-9261-44c2-bc78-54bd30e83dea\") "
Oct 30 20:10:31.945604 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.945583    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2982e319-9261-44c2-bc78-54bd30e83dea-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:31.946904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:31.946882    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2982e319-9261-44c2-bc78-54bd30e83dea-kube-api-access-2cfj4" pod "2982e319-9261-44c2-bc78-54bd30e83dea" (UID: "2982e319-9261-44c2-bc78-54bd30e83dea"). InnerVolumeSpecName "kube-api-access-2cfj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:32.007300 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.007266    5408 generic.go:356] "Generic (PLEG): container finished" podID="cc5cea2d-0de3-445f-923c-44341da06bc8" containerID="b08475cfcb3149f00d87dcd7a52dc6df71193984c118ed753332b51f80be1b21" exitCode=0
Oct 30 20:10:32.007490 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.007326    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"
Oct 30 20:10:32.007490 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.007367    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz" event={"ID":"cc5cea2d-0de3-445f-923c-44341da06bc8","Type":"ContainerDied","Data":"b08475cfcb3149f00d87dcd7a52dc6df71193984c118ed753332b51f80be1b21"}
Oct 30 20:10:32.007490 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.007411    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz" event={"ID":"cc5cea2d-0de3-445f-923c-44341da06bc8","Type":"ContainerDied","Data":"3ac7ebcd0ce8aae4e8b06ab3fad16693d545cf43152d564ab99d50e242b51a3e"}
Oct 30 20:10:32.007490 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.007432    5408 scope.go:122] "RemoveContainer" containerID="b08475cfcb3149f00d87dcd7a52dc6df71193984c118ed753332b51f80be1b21"
Oct 30 20:10:32.012319 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.012292    5408 generic.go:356] "Generic (PLEG): container finished" podID="2982e319-9261-44c2-bc78-54bd30e83dea" containerID="29bb6b68d12dab06f4b98d3a8ed4f1dd405a07d8f6afd96204bbf5bd314e89ec" exitCode=0
Oct 30 20:10:32.012481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.012337    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg"
Oct 30 20:10:32.012554 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.012527    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg" event={"ID":"2982e319-9261-44c2-bc78-54bd30e83dea","Type":"ContainerDied","Data":"29bb6b68d12dab06f4b98d3a8ed4f1dd405a07d8f6afd96204bbf5bd314e89ec"}
Oct 30 20:10:32.012632 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.012609    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg" event={"ID":"2982e319-9261-44c2-bc78-54bd30e83dea","Type":"ContainerDied","Data":"ea53f7acf22e67c726ab5e5601aab70cd222e196052fc1ce7e5c749d33ca7ac1"}
Oct 30 20:10:32.036299 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.036277    5408 scope.go:122] "RemoveContainer" containerID="b08475cfcb3149f00d87dcd7a52dc6df71193984c118ed753332b51f80be1b21"
Oct 30 20:10:32.036678 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:10:32.036645    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08475cfcb3149f00d87dcd7a52dc6df71193984c118ed753332b51f80be1b21\": container with ID starting with b08475cfcb3149f00d87dcd7a52dc6df71193984c118ed753332b51f80be1b21 not found: ID does not exist" containerID="b08475cfcb3149f00d87dcd7a52dc6df71193984c118ed753332b51f80be1b21"
Oct 30 20:10:32.036772 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.036680    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08475cfcb3149f00d87dcd7a52dc6df71193984c118ed753332b51f80be1b21"} err="failed to get container status \"b08475cfcb3149f00d87dcd7a52dc6df71193984c118ed753332b51f80be1b21\": rpc error: code = NotFound desc = could not find container \"b08475cfcb3149f00d87dcd7a52dc6df71193984c118ed753332b51f80be1b21\": container with ID starting with b08475cfcb3149f00d87dcd7a52dc6df71193984c118ed753332b51f80be1b21 not found: ID does not exist"
Oct 30 20:10:32.036772 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.036698    5408 scope.go:122] "RemoveContainer" containerID="29bb6b68d12dab06f4b98d3a8ed4f1dd405a07d8f6afd96204bbf5bd314e89ec"
Oct 30 20:10:32.044208 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.044181    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"]
Oct 30 20:10:32.051210 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.051187    5408 scope.go:122] "RemoveContainer" containerID="29bb6b68d12dab06f4b98d3a8ed4f1dd405a07d8f6afd96204bbf5bd314e89ec"
Oct 30 20:10:32.051359 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.051330    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6986a4/test-model-6986a4-kserve-7b9b7c5567-n27vz"]
Oct 30 20:10:32.051815 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:10:32.051784    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29bb6b68d12dab06f4b98d3a8ed4f1dd405a07d8f6afd96204bbf5bd314e89ec\": container with ID starting with 29bb6b68d12dab06f4b98d3a8ed4f1dd405a07d8f6afd96204bbf5bd314e89ec not found: ID does not exist" containerID="29bb6b68d12dab06f4b98d3a8ed4f1dd405a07d8f6afd96204bbf5bd314e89ec"
Oct 30 20:10:32.051918 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.051820    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29bb6b68d12dab06f4b98d3a8ed4f1dd405a07d8f6afd96204bbf5bd314e89ec"} err="failed to get container status \"29bb6b68d12dab06f4b98d3a8ed4f1dd405a07d8f6afd96204bbf5bd314e89ec\": rpc error: code = NotFound desc = could not find container \"29bb6b68d12dab06f4b98d3a8ed4f1dd405a07d8f6afd96204bbf5bd314e89ec\": container with ID starting with 29bb6b68d12dab06f4b98d3a8ed4f1dd405a07d8f6afd96204bbf5bd314e89ec not found: ID does not exist"
Oct 30 20:10:32.052552 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.052532    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cfj4\" (UniqueName: \"kubernetes.io/projected/2982e319-9261-44c2-bc78-54bd30e83dea-kube-api-access-2cfj4\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:32.072347 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.072324    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg"]
Oct 30 20:10:32.080956 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:32.080931    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7f7d21-7db765c5dc-hh2rg"]
Oct 30 20:10:33.535181 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:33.535154    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2982e319-9261-44c2-bc78-54bd30e83dea" path="/var/lib/kubelet/pods/2982e319-9261-44c2-bc78-54bd30e83dea/volumes"
Oct 30 20:10:33.535671 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:33.535657    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc5cea2d-0de3-445f-923c-44341da06bc8" path="/var/lib/kubelet/pods/cc5cea2d-0de3-445f-923c-44341da06bc8/volumes"
Oct 30 20:10:36.201415 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.201381    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7" podUID="ac13574a-40f3-4228-affd-51573e0296c1" containerName="main" containerID="cri-o://7c4ba5eecef01a5068afbb5515bcc7d89c5389ba122b6dfa707df5b0ecedd05a" gracePeriod=45
Oct 30 20:10:36.530045 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.530006    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:10:36.581498 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.581462    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-home\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-home\") pod \"ac13574a-40f3-4228-affd-51573e0296c1\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") "
Oct 30 20:10:36.581679 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.581534    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac13574a-40f3-4228-affd-51573e0296c1-kube-api-access-w98dm\" (UniqueName: \"kubernetes.io/projected/ac13574a-40f3-4228-affd-51573e0296c1-kube-api-access-w98dm\") pod \"ac13574a-40f3-4228-affd-51573e0296c1\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") "
Oct 30 20:10:36.581679 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.581572    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-model-cache\") pod \"ac13574a-40f3-4228-affd-51573e0296c1\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") "
Oct 30 20:10:36.581679 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.581649    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac13574a-40f3-4228-affd-51573e0296c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac13574a-40f3-4228-affd-51573e0296c1-tls-certs\") pod \"ac13574a-40f3-4228-affd-51573e0296c1\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") "
Oct 30 20:10:36.581873 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.581796    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-home" pod "ac13574a-40f3-4228-affd-51573e0296c1" (UID: "ac13574a-40f3-4228-affd-51573e0296c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:36.581934 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.581889    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-model-cache" pod "ac13574a-40f3-4228-affd-51573e0296c1" (UID: "ac13574a-40f3-4228-affd-51573e0296c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:36.582092 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.582070    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-tmp-dir\") pod \"ac13574a-40f3-4228-affd-51573e0296c1\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") "
Oct 30 20:10:36.582358 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.582333    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-dshm\") pod \"ac13574a-40f3-4228-affd-51573e0296c1\" (UID: \"ac13574a-40f3-4228-affd-51573e0296c1\") "
Oct 30 20:10:36.582681 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.582645    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-tmp-dir" pod "ac13574a-40f3-4228-affd-51573e0296c1" (UID: "ac13574a-40f3-4228-affd-51573e0296c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:36.583231 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.583184    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:36.583231 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.583208    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:36.583231 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.583225    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:36.584327 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.584303    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac13574a-40f3-4228-affd-51573e0296c1-kube-api-access-w98dm" pod "ac13574a-40f3-4228-affd-51573e0296c1" (UID: "ac13574a-40f3-4228-affd-51573e0296c1"). InnerVolumeSpecName "kube-api-access-w98dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:36.584463 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.584439    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac13574a-40f3-4228-affd-51573e0296c1-tls-certs" pod "ac13574a-40f3-4228-affd-51573e0296c1" (UID: "ac13574a-40f3-4228-affd-51573e0296c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:36.584566 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.584550    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-dshm" pod "ac13574a-40f3-4228-affd-51573e0296c1" (UID: "ac13574a-40f3-4228-affd-51573e0296c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:36.693652 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.693597    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac13574a-40f3-4228-affd-51573e0296c1-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:36.693652 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.693635    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w98dm\" (UniqueName: \"kubernetes.io/projected/ac13574a-40f3-4228-affd-51573e0296c1-kube-api-access-w98dm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:36.693652 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.693651    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac13574a-40f3-4228-affd-51573e0296c1-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:36.694653 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:36.694623    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"]
Oct 30 20:10:37.102076 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:37.102019    5408 generic.go:356] "Generic (PLEG): container finished" podID="ac13574a-40f3-4228-affd-51573e0296c1" containerID="7c4ba5eecef01a5068afbb5515bcc7d89c5389ba122b6dfa707df5b0ecedd05a" exitCode=0
Oct 30 20:10:37.102353 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:37.102329    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7" event={"ID":"ac13574a-40f3-4228-affd-51573e0296c1","Type":"ContainerDied","Data":"7c4ba5eecef01a5068afbb5515bcc7d89c5389ba122b6dfa707df5b0ecedd05a"}
Oct 30 20:10:37.102587 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:37.102567    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7" event={"ID":"ac13574a-40f3-4228-affd-51573e0296c1","Type":"ContainerDied","Data":"33421ed289b23f1a6b9658ebc50c4f073343d00b2bffe79d1988989c51c83e9a"}
Oct 30 20:10:37.102727 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:37.102512    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"
Oct 30 20:10:37.103072 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:37.102692    5408 scope.go:122] "RemoveContainer" containerID="7c4ba5eecef01a5068afbb5515bcc7d89c5389ba122b6dfa707df5b0ecedd05a"
Oct 30 20:10:37.127815 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:37.127787    5408 scope.go:122] "RemoveContainer" containerID="7c4ba5eecef01a5068afbb5515bcc7d89c5389ba122b6dfa707df5b0ecedd05a"
Oct 30 20:10:37.128435 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:10:37.128395    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c4ba5eecef01a5068afbb5515bcc7d89c5389ba122b6dfa707df5b0ecedd05a\": container with ID starting with 7c4ba5eecef01a5068afbb5515bcc7d89c5389ba122b6dfa707df5b0ecedd05a not found: ID does not exist" containerID="7c4ba5eecef01a5068afbb5515bcc7d89c5389ba122b6dfa707df5b0ecedd05a"
Oct 30 20:10:37.128613 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:37.128583    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4ba5eecef01a5068afbb5515bcc7d89c5389ba122b6dfa707df5b0ecedd05a"} err="failed to get container status \"7c4ba5eecef01a5068afbb5515bcc7d89c5389ba122b6dfa707df5b0ecedd05a\": rpc error: code = NotFound desc = could not find container \"7c4ba5eecef01a5068afbb5515bcc7d89c5389ba122b6dfa707df5b0ecedd05a\": container with ID starting with 7c4ba5eecef01a5068afbb5515bcc7d89c5389ba122b6dfa707df5b0ecedd05a not found: ID does not exist"
Oct 30 20:10:37.199212 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:37.199179    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"]
Oct 30 20:10:37.203878 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:37.203847    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7"]
Oct 30 20:10:37.486190 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:37.486109    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-55d2e0/test-model-55d2e0-kserve-59bb5dfcb5-tn6b7" podUID="ac13574a-40f3-4228-affd-51573e0296c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.163:8000/ready\": context deadline exceeded"
Oct 30 20:10:37.535774 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:37.535738    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac13574a-40f3-4228-affd-51573e0296c1" path="/var/lib/kubelet/pods/ac13574a-40f3-4228-affd-51573e0296c1/volumes"
Oct 30 20:10:39.667299 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:10:39.667261    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:10:41.372245 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:41.372213    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb"]
Oct 30 20:10:41.990833 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:41.984587    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh" podUID="933df3f9-ffba-40a6-b455-4ddf13776704" containerName="main" containerID="cri-o://0923206460aa67d84e454170a730517cda7e113a6455c35905e084c8e8705e92" gracePeriod=45
Oct 30 20:10:42.126185 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.126146    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh" podUID="933df3f9-ffba-40a6-b455-4ddf13776704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.150:8000/ready\": dial tcp 10.134.0.150:8000: connect: connection refused"
Oct 30 20:10:42.186445 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.186410    5408 generic.go:356] "Generic (PLEG): container finished" podID="933df3f9-ffba-40a6-b455-4ddf13776704" containerID="0923206460aa67d84e454170a730517cda7e113a6455c35905e084c8e8705e92" exitCode=0
Oct 30 20:10:42.186644 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.186627    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh" event={"ID":"933df3f9-ffba-40a6-b455-4ddf13776704","Type":"ContainerDied","Data":"0923206460aa67d84e454170a730517cda7e113a6455c35905e084c8e8705e92"}
Oct 30 20:10:42.307614 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.307589    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:10:42.450421 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.450381    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/933df3f9-ffba-40a6-b455-4ddf13776704-tls-certs\" (UniqueName: \"kubernetes.io/secret/933df3f9-ffba-40a6-b455-4ddf13776704-tls-certs\") pod \"933df3f9-ffba-40a6-b455-4ddf13776704\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") "
Oct 30 20:10:42.450827 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.450470    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-model-cache\") pod \"933df3f9-ffba-40a6-b455-4ddf13776704\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") "
Oct 30 20:10:42.450827 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.450639    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933df3f9-ffba-40a6-b455-4ddf13776704-kube-api-access-97f87\" (UniqueName: \"kubernetes.io/projected/933df3f9-ffba-40a6-b455-4ddf13776704-kube-api-access-97f87\") pod \"933df3f9-ffba-40a6-b455-4ddf13776704\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") "
Oct 30 20:10:42.450827 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.450729    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-home\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-home\") pod \"933df3f9-ffba-40a6-b455-4ddf13776704\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") "
Oct 30 20:10:42.450827 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.450780    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-model-cache" pod "933df3f9-ffba-40a6-b455-4ddf13776704" (UID: "933df3f9-ffba-40a6-b455-4ddf13776704"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:42.450827 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.450799    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-tmp-dir\") pod \"933df3f9-ffba-40a6-b455-4ddf13776704\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") "
Oct 30 20:10:42.450992 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.450875    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-dshm\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-dshm\") pod \"933df3f9-ffba-40a6-b455-4ddf13776704\" (UID: \"933df3f9-ffba-40a6-b455-4ddf13776704\") "
Oct 30 20:10:42.450992 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.450974    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-home" pod "933df3f9-ffba-40a6-b455-4ddf13776704" (UID: "933df3f9-ffba-40a6-b455-4ddf13776704"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:42.451111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.451060    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-tmp-dir" pod "933df3f9-ffba-40a6-b455-4ddf13776704" (UID: "933df3f9-ffba-40a6-b455-4ddf13776704"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:42.451715 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.451691    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:42.451715 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.451713    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:42.451899 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.451723    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:42.452781 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.452759    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/933df3f9-ffba-40a6-b455-4ddf13776704-tls-certs" pod "933df3f9-ffba-40a6-b455-4ddf13776704" (UID: "933df3f9-ffba-40a6-b455-4ddf13776704"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:42.452942 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.452924    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-dshm" pod "933df3f9-ffba-40a6-b455-4ddf13776704" (UID: "933df3f9-ffba-40a6-b455-4ddf13776704"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:42.453066 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.453007    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933df3f9-ffba-40a6-b455-4ddf13776704-kube-api-access-97f87" pod "933df3f9-ffba-40a6-b455-4ddf13776704" (UID: "933df3f9-ffba-40a6-b455-4ddf13776704"). InnerVolumeSpecName "kube-api-access-97f87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:42.553518 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.553435    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97f87\" (UniqueName: \"kubernetes.io/projected/933df3f9-ffba-40a6-b455-4ddf13776704-kube-api-access-97f87\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:42.553518 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.553473    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/933df3f9-ffba-40a6-b455-4ddf13776704-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:42.553518 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:42.553485    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933df3f9-ffba-40a6-b455-4ddf13776704-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:43.195519 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:43.195483    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh" event={"ID":"933df3f9-ffba-40a6-b455-4ddf13776704","Type":"ContainerDied","Data":"c9c02aa53084109039019a847991a78b633fb47037ac6aa55eff0ea469ad1962"}
Oct 30 20:10:43.195717 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:43.195534    5408 scope.go:122] "RemoveContainer" containerID="0923206460aa67d84e454170a730517cda7e113a6455c35905e084c8e8705e92"
Oct 30 20:10:43.195717 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:43.195532    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"
Oct 30 20:10:43.224876 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:43.224844    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"]
Oct 30 20:10:43.230776 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:43.230748    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1f246b/test-model-1f246b-kserve-7497b49b9c-j2vzh"]
Oct 30 20:10:43.535167 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:43.535140    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="933df3f9-ffba-40a6-b455-4ddf13776704" path="/var/lib/kubelet/pods/933df3f9-ffba-40a6-b455-4ddf13776704/volumes"
Oct 30 20:10:45.426078 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:45.426046    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4"]
Oct 30 20:10:45.431934 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:45.431909    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7f7d21-fxlh4"]
Oct 30 20:10:45.578054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:45.570017    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a25adf-6a0b-44fe-a31b-b85b1d44fc7d" path="/var/lib/kubelet/pods/07a25adf-6a0b-44fe-a31b-b85b1d44fc7d/volumes"
Oct 30 20:10:46.373383 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:46.373327    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb" podUID="f877a279-f254-4591-a1ee-fa1804d8566b" containerName="payload-processing" containerID="cri-o://0d8f67ef01e9803c08bae4de130203968cbac10435d4d9e34a42bcedc4ed4762" gracePeriod=25
Oct 30 20:10:47.018915 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:47.018882    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb"
Oct 30 20:10:47.147737 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:47.147700    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f877a279-f254-4591-a1ee-fa1804d8566b-kube-api-access-466qk\" (UniqueName: \"kubernetes.io/projected/f877a279-f254-4591-a1ee-fa1804d8566b-kube-api-access-466qk\") pod \"f877a279-f254-4591-a1ee-fa1804d8566b\" (UID: \"f877a279-f254-4591-a1ee-fa1804d8566b\") "
Oct 30 20:10:47.148211 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:47.148182    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f877a279-f254-4591-a1ee-fa1804d8566b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f877a279-f254-4591-a1ee-fa1804d8566b-plugins-config-volume\") pod \"f877a279-f254-4591-a1ee-fa1804d8566b\" (UID: \"f877a279-f254-4591-a1ee-fa1804d8566b\") "
Oct 30 20:10:47.149266 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:47.149223    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f877a279-f254-4591-a1ee-fa1804d8566b-plugins-config-volume" pod "f877a279-f254-4591-a1ee-fa1804d8566b" (UID: "f877a279-f254-4591-a1ee-fa1804d8566b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:47.150166 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:47.150143    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f877a279-f254-4591-a1ee-fa1804d8566b-kube-api-access-466qk" pod "f877a279-f254-4591-a1ee-fa1804d8566b" (UID: "f877a279-f254-4591-a1ee-fa1804d8566b"). InnerVolumeSpecName "kube-api-access-466qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:47.254651 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:47.254609    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f877a279-f254-4591-a1ee-fa1804d8566b-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:47.254651 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:47.254651    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-466qk\" (UniqueName: \"kubernetes.io/projected/f877a279-f254-4591-a1ee-fa1804d8566b-kube-api-access-466qk\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:47.267397 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:47.267362    5408 generic.go:356] "Generic (PLEG): container finished" podID="f877a279-f254-4591-a1ee-fa1804d8566b" containerID="0d8f67ef01e9803c08bae4de130203968cbac10435d4d9e34a42bcedc4ed4762" exitCode=0
Oct 30 20:10:47.267581 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:47.267406    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb"
Oct 30 20:10:47.267655 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:47.267583    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb" event={"ID":"f877a279-f254-4591-a1ee-fa1804d8566b","Type":"ContainerDied","Data":"0d8f67ef01e9803c08bae4de130203968cbac10435d4d9e34a42bcedc4ed4762"}
Oct 30 20:10:47.267720 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:47.267694    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb" event={"ID":"f877a279-f254-4591-a1ee-fa1804d8566b","Type":"ContainerDied","Data":"693001d51c78fd113a51235d488fb176ea4713d9ad39d80af4c42fadc3d65190"}
Oct 30 20:10:47.267785 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:47.267726    5408 scope.go:122] "RemoveContainer" containerID="0d8f67ef01e9803c08bae4de130203968cbac10435d4d9e34a42bcedc4ed4762"
Oct 30 20:10:47.281296 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:47.281275    5408 scope.go:122] "RemoveContainer" containerID="0d8f67ef01e9803c08bae4de130203968cbac10435d4d9e34a42bcedc4ed4762"
Oct 30 20:10:47.281745 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:10:47.281606    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8f67ef01e9803c08bae4de130203968cbac10435d4d9e34a42bcedc4ed4762\": container with ID starting with 0d8f67ef01e9803c08bae4de130203968cbac10435d4d9e34a42bcedc4ed4762 not found: ID does not exist" containerID="0d8f67ef01e9803c08bae4de130203968cbac10435d4d9e34a42bcedc4ed4762"
Oct 30 20:10:47.281745 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:47.281638    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8f67ef01e9803c08bae4de130203968cbac10435d4d9e34a42bcedc4ed4762"} err="failed to get container status \"0d8f67ef01e9803c08bae4de130203968cbac10435d4d9e34a42bcedc4ed4762\": rpc error: code = NotFound desc = could not find container \"0d8f67ef01e9803c08bae4de130203968cbac10435d4d9e34a42bcedc4ed4762\": container with ID starting with 0d8f67ef01e9803c08bae4de130203968cbac10435d4d9e34a42bcedc4ed4762 not found: ID does not exist"
Oct 30 20:10:47.333170 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:47.333131    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb"]
Oct 30 20:10:47.334663 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:47.334634    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f5eee-6f889d6cdb-khszb"]
Oct 30 20:10:47.590989 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:47.590900    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f877a279-f254-4591-a1ee-fa1804d8566b" path="/var/lib/kubelet/pods/f877a279-f254-4591-a1ee-fa1804d8566b/volumes"
Oct 30 20:10:51.722956 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:51.722900    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls" podUID="f8b220f9-8403-4d34-a940-200f160de17f" containerName="main" containerID="cri-o://b80f917f8e6390feae84b3beac0c737c7ea5732411d7dbfe4eeca22579200808" gracePeriod=45
Oct 30 20:10:51.734545 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:51.734507    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls" podUID="f8b220f9-8403-4d34-a940-200f160de17f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.153:8000/ready\": dial tcp 10.134.0.153:8000: connect: connection refused"
Oct 30 20:10:52.136203 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.136177    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:10:52.277265 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.277230    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8b220f9-8403-4d34-a940-200f160de17f-tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b220f9-8403-4d34-a940-200f160de17f-tls-certs\") pod \"f8b220f9-8403-4d34-a940-200f160de17f\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") "
Oct 30 20:10:52.277475 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.277287    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-model-cache\") pod \"f8b220f9-8403-4d34-a940-200f160de17f\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") "
Oct 30 20:10:52.277475 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.277381    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-tmp-dir\") pod \"f8b220f9-8403-4d34-a940-200f160de17f\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") "
Oct 30 20:10:52.277601 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.277576    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-model-cache" pod "f8b220f9-8403-4d34-a940-200f160de17f" (UID: "f8b220f9-8403-4d34-a940-200f160de17f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:52.277601 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.277591    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-home\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-home\") pod \"f8b220f9-8403-4d34-a940-200f160de17f\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") "
Oct 30 20:10:52.277713 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.277627    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-tmp-dir" pod "f8b220f9-8403-4d34-a940-200f160de17f" (UID: "f8b220f9-8403-4d34-a940-200f160de17f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:52.277713 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.277639    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b220f9-8403-4d34-a940-200f160de17f-kube-api-access-v5xm9\" (UniqueName: \"kubernetes.io/projected/f8b220f9-8403-4d34-a940-200f160de17f-kube-api-access-v5xm9\") pod \"f8b220f9-8403-4d34-a940-200f160de17f\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") "
Oct 30 20:10:52.277713 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.277685    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-dshm\") pod \"f8b220f9-8403-4d34-a940-200f160de17f\" (UID: \"f8b220f9-8403-4d34-a940-200f160de17f\") "
Oct 30 20:10:52.277880 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.277811    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-home" pod "f8b220f9-8403-4d34-a940-200f160de17f" (UID: "f8b220f9-8403-4d34-a940-200f160de17f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:52.278586 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.278560    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:52.278721 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.278589    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:52.278721 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.278604    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:52.279736 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.279712    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8b220f9-8403-4d34-a940-200f160de17f-tls-certs" pod "f8b220f9-8403-4d34-a940-200f160de17f" (UID: "f8b220f9-8403-4d34-a940-200f160de17f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:52.279736 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.279727    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-dshm" pod "f8b220f9-8403-4d34-a940-200f160de17f" (UID: "f8b220f9-8403-4d34-a940-200f160de17f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:52.279852 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.279810    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b220f9-8403-4d34-a940-200f160de17f-kube-api-access-v5xm9" pod "f8b220f9-8403-4d34-a940-200f160de17f" (UID: "f8b220f9-8403-4d34-a940-200f160de17f"). InnerVolumeSpecName "kube-api-access-v5xm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:52.311698 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.311667    5408 generic.go:356] "Generic (PLEG): container finished" podID="f8b220f9-8403-4d34-a940-200f160de17f" containerID="b80f917f8e6390feae84b3beac0c737c7ea5732411d7dbfe4eeca22579200808" exitCode=0
Oct 30 20:10:52.311842 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.311721    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"
Oct 30 20:10:52.311842 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.311757    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls" event={"ID":"f8b220f9-8403-4d34-a940-200f160de17f","Type":"ContainerDied","Data":"b80f917f8e6390feae84b3beac0c737c7ea5732411d7dbfe4eeca22579200808"}
Oct 30 20:10:52.311842 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.311807    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls" event={"ID":"f8b220f9-8403-4d34-a940-200f160de17f","Type":"ContainerDied","Data":"df50d14ef0012ccf7476e13b17b9d2dde94678c9b8070d3517deba6145c69ace"}
Oct 30 20:10:52.311842 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.311829    5408 scope.go:122] "RemoveContainer" containerID="b80f917f8e6390feae84b3beac0c737c7ea5732411d7dbfe4eeca22579200808"
Oct 30 20:10:52.323622 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.323605    5408 scope.go:122] "RemoveContainer" containerID="b80f917f8e6390feae84b3beac0c737c7ea5732411d7dbfe4eeca22579200808"
Oct 30 20:10:52.323856 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:10:52.323840    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b80f917f8e6390feae84b3beac0c737c7ea5732411d7dbfe4eeca22579200808\": container with ID starting with b80f917f8e6390feae84b3beac0c737c7ea5732411d7dbfe4eeca22579200808 not found: ID does not exist" containerID="b80f917f8e6390feae84b3beac0c737c7ea5732411d7dbfe4eeca22579200808"
Oct 30 20:10:52.323912 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.323863    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b80f917f8e6390feae84b3beac0c737c7ea5732411d7dbfe4eeca22579200808"} err="failed to get container status \"b80f917f8e6390feae84b3beac0c737c7ea5732411d7dbfe4eeca22579200808\": rpc error: code = NotFound desc = could not find container \"b80f917f8e6390feae84b3beac0c737c7ea5732411d7dbfe4eeca22579200808\": container with ID starting with b80f917f8e6390feae84b3beac0c737c7ea5732411d7dbfe4eeca22579200808 not found: ID does not exist"
Oct 30 20:10:52.339763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.339737    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"]
Oct 30 20:10:52.343420 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.343398    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-db02a7/test-model-db02a7-kserve-66fcd4c6bc-lf5ls"]
Oct 30 20:10:52.380017 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.379962    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5xm9\" (UniqueName: \"kubernetes.io/projected/f8b220f9-8403-4d34-a940-200f160de17f-kube-api-access-v5xm9\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:52.380017 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.379986    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8b220f9-8403-4d34-a940-200f160de17f-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:52.380017 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:52.379999    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b220f9-8403-4d34-a940-200f160de17f-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:53.535855 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:53.535819    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8b220f9-8403-4d34-a940-200f160de17f" path="/var/lib/kubelet/pods/f8b220f9-8403-4d34-a940-200f160de17f/volumes"
Oct 30 20:10:53.605072 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:53.603022    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"]
Oct 30 20:10:53.605072 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:53.603259    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2" podUID="7c77d77b-fee8-48c4-a717-5c17f9f02453" containerName="istio-proxy" containerID="cri-o://ec614a3f56d6fa54ddd1dbeb4d05fef36074af35e9f4a0078ba28cf86c47a9f6" gracePeriod=30
Oct 30 20:10:55.124135 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:55.124096    5408 patch_prober.go:29] interesting pod/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:10:55.124746 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:55.124173    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2" podUID="7c77d77b-fee8-48c4-a717-5c17f9f02453" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:10:58.743713 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:58.743678    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"]
Oct 30 20:10:58.744210 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:58.743924    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk" podUID="16a79c80-771e-418e-80ca-af817df5d495" containerName="maas-api" containerID="cri-o://bc5a0962e9520d18afe3e8693e17c591b68fcd650dab209b0b1f9d7f0af9cfaf" gracePeriod=30
Oct 30 20:10:59.100052 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.097209    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:10:59.227112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.172397    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-workload-socket\") pod \"7c77d77b-fee8-48c4-a717-5c17f9f02453\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") "
Oct 30 20:10:59.227112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.172462    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-data\") pod \"7c77d77b-fee8-48c4-a717-5c17f9f02453\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") "
Oct 30 20:10:59.227112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.172550    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-credential-socket\") pod \"7c77d77b-fee8-48c4-a717-5c17f9f02453\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") "
Oct 30 20:10:59.227112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.172610    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-envoy\") pod \"7c77d77b-fee8-48c4-a717-5c17f9f02453\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") "
Oct 30 20:10:59.227112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.172666    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c77d77b-fee8-48c4-a717-5c17f9f02453-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c77d77b-fee8-48c4-a717-5c17f9f02453-istiod-ca-cert\") pod \"7c77d77b-fee8-48c4-a717-5c17f9f02453\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") "
Oct 30 20:10:59.227112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.172816    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-podinfo\") pod \"7c77d77b-fee8-48c4-a717-5c17f9f02453\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") "
Oct 30 20:10:59.227112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.172908    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-workload-certs\") pod \"7c77d77b-fee8-48c4-a717-5c17f9f02453\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") "
Oct 30 20:10:59.227112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.173138    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-token\" (UniqueName: \"kubernetes.io/projected/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-token\") pod \"7c77d77b-fee8-48c4-a717-5c17f9f02453\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") "
Oct 30 20:10:59.227112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.173192    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c77d77b-fee8-48c4-a717-5c17f9f02453-kube-api-access-z92sv\" (UniqueName: \"kubernetes.io/projected/7c77d77b-fee8-48c4-a717-5c17f9f02453-kube-api-access-z92sv\") pod \"7c77d77b-fee8-48c4-a717-5c17f9f02453\" (UID: \"7c77d77b-fee8-48c4-a717-5c17f9f02453\") "
Oct 30 20:10:59.227112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.191154    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c77d77b-fee8-48c4-a717-5c17f9f02453-istiod-ca-cert" pod "7c77d77b-fee8-48c4-a717-5c17f9f02453" (UID: "7c77d77b-fee8-48c4-a717-5c17f9f02453"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:59.227112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.191470    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-workload-socket" pod "7c77d77b-fee8-48c4-a717-5c17f9f02453" (UID: "7c77d77b-fee8-48c4-a717-5c17f9f02453"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:59.227112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.191639    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-data" pod "7c77d77b-fee8-48c4-a717-5c17f9f02453" (UID: "7c77d77b-fee8-48c4-a717-5c17f9f02453"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:59.227112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.191798    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-workload-certs" pod "7c77d77b-fee8-48c4-a717-5c17f9f02453" (UID: "7c77d77b-fee8-48c4-a717-5c17f9f02453"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:59.227112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.191973    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-credential-socket" pod "7c77d77b-fee8-48c4-a717-5c17f9f02453" (UID: "7c77d77b-fee8-48c4-a717-5c17f9f02453"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:59.241056 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.240380    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-token" pod "7c77d77b-fee8-48c4-a717-5c17f9f02453" (UID: "7c77d77b-fee8-48c4-a717-5c17f9f02453"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:59.241056 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.240533    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-podinfo" pod "7c77d77b-fee8-48c4-a717-5c17f9f02453" (UID: "7c77d77b-fee8-48c4-a717-5c17f9f02453"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:10:59.252131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.246675    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c77d77b-fee8-48c4-a717-5c17f9f02453-kube-api-access-z92sv" pod "7c77d77b-fee8-48c4-a717-5c17f9f02453" (UID: "7c77d77b-fee8-48c4-a717-5c17f9f02453"). InnerVolumeSpecName "kube-api-access-z92sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:59.252131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.248633    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-envoy" pod "7c77d77b-fee8-48c4-a717-5c17f9f02453" (UID: "7c77d77b-fee8-48c4-a717-5c17f9f02453"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:59.319055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.317643    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:59.319055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.317686    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z92sv\" (UniqueName: \"kubernetes.io/projected/7c77d77b-fee8-48c4-a717-5c17f9f02453-kube-api-access-z92sv\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:59.319055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.317705    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:59.319055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.317719    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:59.319055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.317734    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:59.319055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.317747    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:59.319055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.317760    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c77d77b-fee8-48c4-a717-5c17f9f02453-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:59.319055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.317775    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c77d77b-fee8-48c4-a717-5c17f9f02453-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:59.319055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.317788    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c77d77b-fee8-48c4-a717-5c17f9f02453-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:59.498899 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.498858    5408 generic.go:356] "Generic (PLEG): container finished" podID="16a79c80-771e-418e-80ca-af817df5d495" containerID="bc5a0962e9520d18afe3e8693e17c591b68fcd650dab209b0b1f9d7f0af9cfaf" exitCode=0
Oct 30 20:10:59.499208 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.499181    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk" event={"ID":"16a79c80-771e-418e-80ca-af817df5d495","Type":"ContainerDied","Data":"bc5a0962e9520d18afe3e8693e17c591b68fcd650dab209b0b1f9d7f0af9cfaf"}
Oct 30 20:10:59.501410 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.501380    5408 generic.go:356] "Generic (PLEG): container finished" podID="7c77d77b-fee8-48c4-a717-5c17f9f02453" containerID="ec614a3f56d6fa54ddd1dbeb4d05fef36074af35e9f4a0078ba28cf86c47a9f6" exitCode=0
Oct 30 20:10:59.501800 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.501784    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"
Oct 30 20:10:59.511047 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.505581    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2" event={"ID":"7c77d77b-fee8-48c4-a717-5c17f9f02453","Type":"ContainerDied","Data":"ec614a3f56d6fa54ddd1dbeb4d05fef36074af35e9f4a0078ba28cf86c47a9f6"}
Oct 30 20:10:59.511047 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.505651    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2" event={"ID":"7c77d77b-fee8-48c4-a717-5c17f9f02453","Type":"ContainerDied","Data":"3be1aa178968b57f84fed8d757ceba9ca54b840e0581fc0fdbffe14a2f6eb946"}
Oct 30 20:10:59.511047 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.505682    5408 scope.go:122] "RemoveContainer" containerID="ec614a3f56d6fa54ddd1dbeb4d05fef36074af35e9f4a0078ba28cf86c47a9f6"
Oct 30 20:10:59.542226 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.542201    5408 scope.go:122] "RemoveContainer" containerID="a74979304917224f6fcabbd5f59699827335bed510c65d93c50424a5699b38f6"
Oct 30 20:10:59.546115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.546090    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"]
Oct 30 20:10:59.551942 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.551912    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7f7d21-openshift-default-5878dc6bd5-j6mh2"]
Oct 30 20:10:59.557713 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.557687    5408 scope.go:122] "RemoveContainer" containerID="ec614a3f56d6fa54ddd1dbeb4d05fef36074af35e9f4a0078ba28cf86c47a9f6"
Oct 30 20:10:59.558082 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:10:59.558022    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec614a3f56d6fa54ddd1dbeb4d05fef36074af35e9f4a0078ba28cf86c47a9f6\": container with ID starting with ec614a3f56d6fa54ddd1dbeb4d05fef36074af35e9f4a0078ba28cf86c47a9f6 not found: ID does not exist" containerID="ec614a3f56d6fa54ddd1dbeb4d05fef36074af35e9f4a0078ba28cf86c47a9f6"
Oct 30 20:10:59.558082 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.558065    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec614a3f56d6fa54ddd1dbeb4d05fef36074af35e9f4a0078ba28cf86c47a9f6"} err="failed to get container status \"ec614a3f56d6fa54ddd1dbeb4d05fef36074af35e9f4a0078ba28cf86c47a9f6\": rpc error: code = NotFound desc = could not find container \"ec614a3f56d6fa54ddd1dbeb4d05fef36074af35e9f4a0078ba28cf86c47a9f6\": container with ID starting with ec614a3f56d6fa54ddd1dbeb4d05fef36074af35e9f4a0078ba28cf86c47a9f6 not found: ID does not exist"
Oct 30 20:10:59.558283 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.558084    5408 scope.go:122] "RemoveContainer" containerID="a74979304917224f6fcabbd5f59699827335bed510c65d93c50424a5699b38f6"
Oct 30 20:10:59.558407 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:10:59.558388    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a74979304917224f6fcabbd5f59699827335bed510c65d93c50424a5699b38f6\": container with ID starting with a74979304917224f6fcabbd5f59699827335bed510c65d93c50424a5699b38f6 not found: ID does not exist" containerID="a74979304917224f6fcabbd5f59699827335bed510c65d93c50424a5699b38f6"
Oct 30 20:10:59.558465 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.558413    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a74979304917224f6fcabbd5f59699827335bed510c65d93c50424a5699b38f6"} err="failed to get container status \"a74979304917224f6fcabbd5f59699827335bed510c65d93c50424a5699b38f6\": rpc error: code = NotFound desc = could not find container \"a74979304917224f6fcabbd5f59699827335bed510c65d93c50424a5699b38f6\": container with ID starting with a74979304917224f6fcabbd5f59699827335bed510c65d93c50424a5699b38f6 not found: ID does not exist"
Oct 30 20:10:59.575323 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.575297    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"
Oct 30 20:10:59.633126 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.633089    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16a79c80-771e-418e-80ca-af817df5d495-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16a79c80-771e-418e-80ca-af817df5d495-metrics-certs\") pod \"16a79c80-771e-418e-80ca-af817df5d495\" (UID: \"16a79c80-771e-418e-80ca-af817df5d495\") "
Oct 30 20:10:59.634080 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.633282    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16a79c80-771e-418e-80ca-af817df5d495-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16a79c80-771e-418e-80ca-af817df5d495-maas-api-tls\") pod \"16a79c80-771e-418e-80ca-af817df5d495\" (UID: \"16a79c80-771e-418e-80ca-af817df5d495\") "
Oct 30 20:10:59.634080 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.633349    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a79c80-771e-418e-80ca-af817df5d495-kube-api-access-k9x7w\" (UniqueName: \"kubernetes.io/projected/16a79c80-771e-418e-80ca-af817df5d495-kube-api-access-k9x7w\") pod \"16a79c80-771e-418e-80ca-af817df5d495\" (UID: \"16a79c80-771e-418e-80ca-af817df5d495\") "
Oct 30 20:10:59.634080 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.633522    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16a79c80-771e-418e-80ca-af817df5d495-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a79c80-771e-418e-80ca-af817df5d495-openshift-service-ca\") pod \"16a79c80-771e-418e-80ca-af817df5d495\" (UID: \"16a79c80-771e-418e-80ca-af817df5d495\") "
Oct 30 20:10:59.634346 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.634285    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16a79c80-771e-418e-80ca-af817df5d495-openshift-service-ca" pod "16a79c80-771e-418e-80ca-af817df5d495" (UID: "16a79c80-771e-418e-80ca-af817df5d495"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:59.636383 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.636351    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a79c80-771e-418e-80ca-af817df5d495-kube-api-access-k9x7w" pod "16a79c80-771e-418e-80ca-af817df5d495" (UID: "16a79c80-771e-418e-80ca-af817df5d495"). InnerVolumeSpecName "kube-api-access-k9x7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:59.636509 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.636468    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16a79c80-771e-418e-80ca-af817df5d495-metrics-certs" pod "16a79c80-771e-418e-80ca-af817df5d495" (UID: "16a79c80-771e-418e-80ca-af817df5d495"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:59.640156 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.637454    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16a79c80-771e-418e-80ca-af817df5d495-maas-api-tls" pod "16a79c80-771e-418e-80ca-af817df5d495" (UID: "16a79c80-771e-418e-80ca-af817df5d495"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:59.735668 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.735636    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16a79c80-771e-418e-80ca-af817df5d495-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:59.735668 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.735669    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9x7w\" (UniqueName: \"kubernetes.io/projected/16a79c80-771e-418e-80ca-af817df5d495-kube-api-access-k9x7w\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:59.735875 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.735686    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a79c80-771e-418e-80ca-af817df5d495-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:59.735875 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:10:59.735698    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16a79c80-771e-418e-80ca-af817df5d495-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:00.511577 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:00.511534    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk" event={"ID":"16a79c80-771e-418e-80ca-af817df5d495","Type":"ContainerDied","Data":"7dac98b4b7de9bb807f90ee02b2755371fed6f7f6c06e59283d197097b5089d0"}
Oct 30 20:11:00.512196 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:00.511613    5408 scope.go:122] "RemoveContainer" containerID="bc5a0962e9520d18afe3e8693e17c591b68fcd650dab209b0b1f9d7f0af9cfaf"
Oct 30 20:11:00.512196 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:00.511550    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"
Oct 30 20:11:00.546416 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:00.546385    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"]
Oct 30 20:11:00.550209 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:00.550181    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5eee-647795dcb7-g82rk"]
Oct 30 20:11:01.533949 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:01.533917    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16a79c80-771e-418e-80ca-af817df5d495" path="/var/lib/kubelet/pods/16a79c80-771e-418e-80ca-af817df5d495/volumes"
Oct 30 20:11:01.534568 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:01.534551    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c77d77b-fee8-48c4-a717-5c17f9f02453" path="/var/lib/kubelet/pods/7c77d77b-fee8-48c4-a717-5c17f9f02453/volumes"
Oct 30 20:11:05.495020 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:05.494986    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"]
Oct 30 20:11:05.495565 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:05.495242    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt" podUID="193b7fcd-4f9f-43d4-b833-e8f690bf5df8" containerName="istio-proxy" containerID="cri-o://982f4b5b99705fe79e75a763604796f3ce9ddc4ddcb0f27435bb4f243607c9a9" gracePeriod=30
Oct 30 20:11:05.627349 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:05.627297    5408 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:11:05.627654 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:05.627625    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt" podUID="193b7fcd-4f9f-43d4-b833-e8f690bf5df8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:11:09.663939 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:11:09.663897    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:11:10.100594 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.100561    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"]
Oct 30 20:11:10.787376 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.787296    5408 generic.go:356] "Generic (PLEG): container finished" podID="193b7fcd-4f9f-43d4-b833-e8f690bf5df8" containerID="982f4b5b99705fe79e75a763604796f3ce9ddc4ddcb0f27435bb4f243607c9a9" exitCode=0
Oct 30 20:11:10.787376 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.787366    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt" event={"ID":"193b7fcd-4f9f-43d4-b833-e8f690bf5df8","Type":"ContainerDied","Data":"982f4b5b99705fe79e75a763604796f3ce9ddc4ddcb0f27435bb4f243607c9a9"}
Oct 30 20:11:10.838653 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.838621    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:11:10.998312 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.998250    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-podinfo\") pod \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") "
Oct 30 20:11:10.998523 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.998457    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-token\" (UniqueName: \"kubernetes.io/projected/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-token\") pod \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") "
Oct 30 20:11:10.998597 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.998530    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istiod-ca-cert\") pod \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") "
Oct 30 20:11:10.998652 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.998599    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-envoy\") pod \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") "
Oct 30 20:11:10.998716 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.998698    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-workload-socket\") pod \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") "
Oct 30 20:11:10.998777 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.998765    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-data\") pod \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") "
Oct 30 20:11:10.998834 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.998798    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-kube-api-access-44f7z\" (UniqueName: \"kubernetes.io/projected/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-kube-api-access-44f7z\") pod \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") "
Oct 30 20:11:10.999273 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.998973    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-workload-socket" pod "193b7fcd-4f9f-43d4-b833-e8f690bf5df8" (UID: "193b7fcd-4f9f-43d4-b833-e8f690bf5df8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:10.999273 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.999006    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-workload-certs\") pod \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") "
Oct 30 20:11:10.999273 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.999089    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-data" pod "193b7fcd-4f9f-43d4-b833-e8f690bf5df8" (UID: "193b7fcd-4f9f-43d4-b833-e8f690bf5df8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:10.999273 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.998980    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istiod-ca-cert" pod "193b7fcd-4f9f-43d4-b833-e8f690bf5df8" (UID: "193b7fcd-4f9f-43d4-b833-e8f690bf5df8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:10.999273 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.999138    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-credential-socket\") pod \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\" (UID: \"193b7fcd-4f9f-43d4-b833-e8f690bf5df8\") "
Oct 30 20:11:10.999273 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.999226    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-workload-certs" pod "193b7fcd-4f9f-43d4-b833-e8f690bf5df8" (UID: "193b7fcd-4f9f-43d4-b833-e8f690bf5df8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:10.999693 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.999449    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-credential-socket" pod "193b7fcd-4f9f-43d4-b833-e8f690bf5df8" (UID: "193b7fcd-4f9f-43d4-b833-e8f690bf5df8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:10.999937 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.999915    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:11.000077 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.999943    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:11.000077 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.999961    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:11.000077 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.999978    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:11.000077 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:10.999992    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:11.000891 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:11.000865    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-podinfo" pod "193b7fcd-4f9f-43d4-b833-e8f690bf5df8" (UID: "193b7fcd-4f9f-43d4-b833-e8f690bf5df8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:11:11.001001 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:11.000945    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-token" pod "193b7fcd-4f9f-43d4-b833-e8f690bf5df8" (UID: "193b7fcd-4f9f-43d4-b833-e8f690bf5df8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:11.001346 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:11.001328    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-envoy" pod "193b7fcd-4f9f-43d4-b833-e8f690bf5df8" (UID: "193b7fcd-4f9f-43d4-b833-e8f690bf5df8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:11.001411 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:11.001329    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-kube-api-access-44f7z" pod "193b7fcd-4f9f-43d4-b833-e8f690bf5df8" (UID: "193b7fcd-4f9f-43d4-b833-e8f690bf5df8"). InnerVolumeSpecName "kube-api-access-44f7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:11.101895 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:11.101808    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:11.101895 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:11.101844    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:11.101895 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:11.101854    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:11.101895 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:11.101864    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44f7z\" (UniqueName: \"kubernetes.io/projected/193b7fcd-4f9f-43d4-b833-e8f690bf5df8-kube-api-access-44f7z\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:11.797961 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:11.797879    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"
Oct 30 20:11:11.797961 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:11.797908    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt" event={"ID":"193b7fcd-4f9f-43d4-b833-e8f690bf5df8","Type":"ContainerDied","Data":"153251e63c7e21e4191f2eb8fddb5c9e9a2fe44656e6e9d681d832df3c39500a"}
Oct 30 20:11:11.798480 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:11.797979    5408 scope.go:122] "RemoveContainer" containerID="982f4b5b99705fe79e75a763604796f3ce9ddc4ddcb0f27435bb4f243607c9a9"
Oct 30 20:11:11.829991 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:11.829957    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"]
Oct 30 20:11:11.839218 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:11.839184    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-qpxwt"]
Oct 30 20:11:12.047058 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:12.038916    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"]
Oct 30 20:11:13.536406 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:13.536373    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="193b7fcd-4f9f-43d4-b833-e8f690bf5df8" path="/var/lib/kubelet/pods/193b7fcd-4f9f-43d4-b833-e8f690bf5df8/volumes"
Oct 30 20:11:13.973901 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:13.973810    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"]
Oct 30 20:11:16.114555 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:16.114515    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2"]
Oct 30 20:11:16.133049 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:16.133003    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p"]
Oct 30 20:11:16.791941 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:16.791901    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"]
Oct 30 20:11:16.795310 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:16.795277    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg" podUID="88493f4b-7ec6-4f72-92df-03fb99d9c612" containerName="istio-proxy" containerID="cri-o://740036cd4ff7c7c2f138fd5a15eea5bf22eb69a7c272e192f8fec8ff9e72da9f" gracePeriod=30
Oct 30 20:11:17.295779 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.295740    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"]
Oct 30 20:11:17.296646 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296626    5408 cpu_manager.go:414] "containerMap: removing container" podUID="7c77d77b-fee8-48c4-a717-5c17f9f02453" containerName="istio-proxy"
Oct 30 20:11:17.296695 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296647    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c77d77b-fee8-48c4-a717-5c17f9f02453" containerName="istio-proxy"
Oct 30 20:11:17.296695 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296663    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f877a279-f254-4591-a1ee-fa1804d8566b" containerName="payload-processing"
Oct 30 20:11:17.296695 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296668    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f877a279-f254-4591-a1ee-fa1804d8566b" containerName="payload-processing"
Oct 30 20:11:17.296695 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296677    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f8b220f9-8403-4d34-a940-200f160de17f" containerName="main"
Oct 30 20:11:17.296695 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296683    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b220f9-8403-4d34-a940-200f160de17f" containerName="main"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296698    5408 cpu_manager.go:414] "containerMap: removing container" podUID="193b7fcd-4f9f-43d4-b833-e8f690bf5df8" containerName="istio-proxy"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296703    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="193b7fcd-4f9f-43d4-b833-e8f690bf5df8" containerName="istio-proxy"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296711    5408 cpu_manager.go:414] "containerMap: removing container" podUID="07a25adf-6a0b-44fe-a31b-b85b1d44fc7d" containerName="revoke-keys"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296716    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a25adf-6a0b-44fe-a31b-b85b1d44fc7d" containerName="revoke-keys"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296725    5408 cpu_manager.go:414] "containerMap: removing container" podUID="795803a2-94c7-4773-b51c-5f84a33a28f7" containerName="istio-proxy"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296730    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795803a2-94c7-4773-b51c-5f84a33a28f7" containerName="istio-proxy"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296736    5408 cpu_manager.go:414] "containerMap: removing container" podUID="4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" containerName="istio-proxy"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296740    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" containerName="istio-proxy"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296748    5408 cpu_manager.go:414] "containerMap: removing container" podUID="cc5cea2d-0de3-445f-923c-44341da06bc8" containerName="main"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296753    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc5cea2d-0de3-445f-923c-44341da06bc8" containerName="main"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296770    5408 cpu_manager.go:414] "containerMap: removing container" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerName="istio-proxy"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296774    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerName="istio-proxy"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296790    5408 cpu_manager.go:414] "containerMap: removing container" podUID="ac13574a-40f3-4228-affd-51573e0296c1" containerName="main"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296794    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac13574a-40f3-4228-affd-51573e0296c1" containerName="main"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296807    5408 cpu_manager.go:414] "containerMap: removing container" podUID="2982e319-9261-44c2-bc78-54bd30e83dea" containerName="payload-pre-processing"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296812    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2982e319-9261-44c2-bc78-54bd30e83dea" containerName="payload-pre-processing"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296826    5408 cpu_manager.go:414] "containerMap: removing container" podUID="933df3f9-ffba-40a6-b455-4ddf13776704" containerName="main"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296831    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933df3f9-ffba-40a6-b455-4ddf13776704" containerName="main"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296839    5408 cpu_manager.go:414] "containerMap: removing container" podUID="16a79c80-771e-418e-80ca-af817df5d495" containerName="maas-api"
Oct 30 20:11:17.296886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296843    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a79c80-771e-418e-80ca-af817df5d495" containerName="maas-api"
Oct 30 20:11:17.297584 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296941    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="16a79c80-771e-418e-80ca-af817df5d495" containerName="maas-api"
Oct 30 20:11:17.297584 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296948    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c77d77b-fee8-48c4-a717-5c17f9f02453" containerName="istio-proxy"
Oct 30 20:11:17.297584 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296956    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="f877a279-f254-4591-a1ee-fa1804d8566b" containerName="payload-processing"
Oct 30 20:11:17.297584 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296964    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b220f9-8403-4d34-a940-200f160de17f" containerName="main"
Oct 30 20:11:17.297584 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296973    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="193b7fcd-4f9f-43d4-b833-e8f690bf5df8" containerName="istio-proxy"
Oct 30 20:11:17.297584 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296978    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a25adf-6a0b-44fe-a31b-b85b1d44fc7d" containerName="revoke-keys"
Oct 30 20:11:17.297584 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296987    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="795803a2-94c7-4773-b51c-5f84a33a28f7" containerName="istio-proxy"
Oct 30 20:11:17.297584 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296992    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a7ec4d2-c450-42d0-aebe-8bbf4b8832fe" containerName="istio-proxy"
Oct 30 20:11:17.297584 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.296998    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc5cea2d-0de3-445f-923c-44341da06bc8" containerName="main"
Oct 30 20:11:17.297584 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.297007    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="a746e113-69a0-4d13-acf1-bb28ba1769b7" containerName="istio-proxy"
Oct 30 20:11:17.297584 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.297015    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac13574a-40f3-4228-affd-51573e0296c1" containerName="main"
Oct 30 20:11:17.297584 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.297021    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="2982e319-9261-44c2-bc78-54bd30e83dea" containerName="payload-pre-processing"
Oct 30 20:11:17.297584 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.297044    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="933df3f9-ffba-40a6-b455-4ddf13776704" containerName="main"
Oct 30 20:11:17.302596 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.302573    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.315987 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.315955    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6986a4-openshift-default-dockercfg-rwt7k\""
Oct 30 20:11:17.345963 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.345694    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"]
Oct 30 20:11:17.485189 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.485151    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-credential-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.485381 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.485266    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-workload-certs\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.485537 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.485512    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istiod-ca-cert\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.485638 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.485616    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-envoy\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.485683 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.485668    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-podinfo\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.485729 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.485701    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-data\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.485850 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.485835    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-token\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.485892 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.485859    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv467\" (UniqueName: \"kubernetes.io/projected/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-kube-api-access-pv467\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.485892 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.485882    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-workload-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.586945 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.586845    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istiod-ca-cert\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.587192 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.586955    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-envoy\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.587192 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.586994    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-podinfo\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.587192 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.587026    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-data\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.587192 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.587124    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-token\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.587192 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.587151    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv467\" (UniqueName: \"kubernetes.io/projected/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-kube-api-access-pv467\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.587502 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.587205    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-workload-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.587502 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.587274    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-credential-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.587981 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.587771    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-workload-certs\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.588547 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.588517    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istiod-ca-cert\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.588656 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.588559    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-data\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.588779 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.588753    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-workload-certs\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.588845 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.588778    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-credential-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.588925 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.588895    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-workload-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.590404 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.590377    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-envoy\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.591326 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.591306    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-podinfo\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.602462 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.602433    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-token\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.604517 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.604493    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv467\" (UniqueName: \"kubernetes.io/projected/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-kube-api-access-pv467\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.623071 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.622990    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:17.947413 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.947378    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"]
Oct 30 20:11:17.948199 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:11:17.948171    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ddd1055_47a1_4c05_bd03_ff3c493f32a7.slice/crio-eebd0cdfed94002524abb96e1d05a58e5ed4fd9747ac2a84a90545e3a2e40c1d WatchSource:0}: Error finding container eebd0cdfed94002524abb96e1d05a58e5ed4fd9747ac2a84a90545e3a2e40c1d: Status 404 returned error can't find the container with id eebd0cdfed94002524abb96e1d05a58e5ed4fd9747ac2a84a90545e3a2e40c1d
Oct 30 20:11:17.951286 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.951252    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:11:17.951365 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.951320    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:11:17.951365 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:17.951349    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:11:18.873220 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:18.873171    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9" event={"ID":"4ddd1055-47a1-4c05-bd03-ff3c493f32a7","Type":"ContainerStarted","Data":"7dc0c20e82b2261a27efba3bb9fabb0e26e4254dbb3737f63de9b0c47c03ae9e"}
Oct 30 20:11:18.873662 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:18.873244    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9" event={"ID":"4ddd1055-47a1-4c05-bd03-ff3c493f32a7","Type":"ContainerStarted","Data":"eebd0cdfed94002524abb96e1d05a58e5ed4fd9747ac2a84a90545e3a2e40c1d"}
Oct 30 20:11:18.933134 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:18.933017    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9" podStartSLOduration=1.9329920710000001 podStartE2EDuration="1.932992071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:18.902306428 +0000 UTC m=+2433.486576057" watchObservedRunningTime="2026-10-30 20:11:18.932992071 +0000 UTC m=+2433.517261688"
Oct 30 20:11:19.274071 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:19.269247    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:11:19.274071 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:19.269319    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg" podUID="88493f4b-7ec6-4f72-92df-03fb99d9c612" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:11:19.628158 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:19.627752    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:19.629982 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:19.629941    5408 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 20:11:19.630161 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:19.630004    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9" podUID="4ddd1055-47a1-4c05-bd03-ff3c493f32a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 20:11:20.630624 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:20.630583    5408 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 20:11:20.631251 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:20.631219    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9" podUID="4ddd1055-47a1-4c05-bd03-ff3c493f32a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 20:11:21.118140 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.118094    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2" podUID="aecc895e-a6d4-4fc6-8e55-92b83f17fcde" containerName="payload-processing" containerID="cri-o://ad580d4ee05705771ef6145063227f407db1caa8dff11e02721c78abf10fff09" gracePeriod=25
Oct 30 20:11:21.135576 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.133396    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p" podUID="651ba59f-3f1f-479d-805e-30a17661c602" containerName="payload-pre-processing" containerID="cri-o://8ab54f0292ee3bf38e728dd38f96b66de6b552caf8974b2e129cfa61c7ff08a5" gracePeriod=25
Oct 30 20:11:21.634679 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.634003    5408 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 20:11:21.634679 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.634439    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9" podUID="4ddd1055-47a1-4c05-bd03-ff3c493f32a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 20:11:21.670079 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.668687    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p"
Oct 30 20:11:21.764437 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.764410    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2"
Oct 30 20:11:21.788850 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.788801    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651ba59f-3f1f-479d-805e-30a17661c602-kube-api-access-8zhjc\" (UniqueName: \"kubernetes.io/projected/651ba59f-3f1f-479d-805e-30a17661c602-kube-api-access-8zhjc\") pod \"651ba59f-3f1f-479d-805e-30a17661c602\" (UID: \"651ba59f-3f1f-479d-805e-30a17661c602\") "
Oct 30 20:11:21.789135 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.789111    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651ba59f-3f1f-479d-805e-30a17661c602-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651ba59f-3f1f-479d-805e-30a17661c602-plugins-config-volume\") pod \"651ba59f-3f1f-479d-805e-30a17661c602\" (UID: \"651ba59f-3f1f-479d-805e-30a17661c602\") "
Oct 30 20:11:21.791131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.791101    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651ba59f-3f1f-479d-805e-30a17661c602-plugins-config-volume" pod "651ba59f-3f1f-479d-805e-30a17661c602" (UID: "651ba59f-3f1f-479d-805e-30a17661c602"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:21.801081 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.801025    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651ba59f-3f1f-479d-805e-30a17661c602-kube-api-access-8zhjc" pod "651ba59f-3f1f-479d-805e-30a17661c602" (UID: "651ba59f-3f1f-479d-805e-30a17661c602"). InnerVolumeSpecName "kube-api-access-8zhjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:21.891294 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.891260    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aecc895e-a6d4-4fc6-8e55-92b83f17fcde-kube-api-access-vmx8l\" (UniqueName: \"kubernetes.io/projected/aecc895e-a6d4-4fc6-8e55-92b83f17fcde-kube-api-access-vmx8l\") pod \"aecc895e-a6d4-4fc6-8e55-92b83f17fcde\" (UID: \"aecc895e-a6d4-4fc6-8e55-92b83f17fcde\") "
Oct 30 20:11:21.891479 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.891457    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aecc895e-a6d4-4fc6-8e55-92b83f17fcde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aecc895e-a6d4-4fc6-8e55-92b83f17fcde-plugins-config-volume\") pod \"aecc895e-a6d4-4fc6-8e55-92b83f17fcde\" (UID: \"aecc895e-a6d4-4fc6-8e55-92b83f17fcde\") "
Oct 30 20:11:21.891833 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.891797    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aecc895e-a6d4-4fc6-8e55-92b83f17fcde-plugins-config-volume" pod "aecc895e-a6d4-4fc6-8e55-92b83f17fcde" (UID: "aecc895e-a6d4-4fc6-8e55-92b83f17fcde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:21.891988 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.891973    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aecc895e-a6d4-4fc6-8e55-92b83f17fcde-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:21.892054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.891990    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zhjc\" (UniqueName: \"kubernetes.io/projected/651ba59f-3f1f-479d-805e-30a17661c602-kube-api-access-8zhjc\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:21.892054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.892000    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651ba59f-3f1f-479d-805e-30a17661c602-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:21.893528 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.893503    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aecc895e-a6d4-4fc6-8e55-92b83f17fcde-kube-api-access-vmx8l" pod "aecc895e-a6d4-4fc6-8e55-92b83f17fcde" (UID: "aecc895e-a6d4-4fc6-8e55-92b83f17fcde"). InnerVolumeSpecName "kube-api-access-vmx8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:21.940834 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.940796    5408 generic.go:356] "Generic (PLEG): container finished" podID="651ba59f-3f1f-479d-805e-30a17661c602" containerID="8ab54f0292ee3bf38e728dd38f96b66de6b552caf8974b2e129cfa61c7ff08a5" exitCode=0
Oct 30 20:11:21.940834 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.940835    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p"
Oct 30 20:11:21.941091 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.940875    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p" event={"ID":"651ba59f-3f1f-479d-805e-30a17661c602","Type":"ContainerDied","Data":"8ab54f0292ee3bf38e728dd38f96b66de6b552caf8974b2e129cfa61c7ff08a5"}
Oct 30 20:11:21.941091 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.940926    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p" event={"ID":"651ba59f-3f1f-479d-805e-30a17661c602","Type":"ContainerDied","Data":"6e3b61162081030c52005f7374a9d1df105973f06531935b4ecf258dedb392b9"}
Oct 30 20:11:21.941091 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.940957    5408 scope.go:122] "RemoveContainer" containerID="8ab54f0292ee3bf38e728dd38f96b66de6b552caf8974b2e129cfa61c7ff08a5"
Oct 30 20:11:21.943135 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.943107    5408 generic.go:356] "Generic (PLEG): container finished" podID="88493f4b-7ec6-4f72-92df-03fb99d9c612" containerID="740036cd4ff7c7c2f138fd5a15eea5bf22eb69a7c272e192f8fec8ff9e72da9f" exitCode=0
Oct 30 20:11:21.943258 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.943150    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg" event={"ID":"88493f4b-7ec6-4f72-92df-03fb99d9c612","Type":"ContainerDied","Data":"740036cd4ff7c7c2f138fd5a15eea5bf22eb69a7c272e192f8fec8ff9e72da9f"}
Oct 30 20:11:21.945195 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.945169    5408 generic.go:356] "Generic (PLEG): container finished" podID="aecc895e-a6d4-4fc6-8e55-92b83f17fcde" containerID="ad580d4ee05705771ef6145063227f407db1caa8dff11e02721c78abf10fff09" exitCode=0
Oct 30 20:11:21.945304 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.945223    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2"
Oct 30 20:11:21.945304 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.945247    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2" event={"ID":"aecc895e-a6d4-4fc6-8e55-92b83f17fcde","Type":"ContainerDied","Data":"ad580d4ee05705771ef6145063227f407db1caa8dff11e02721c78abf10fff09"}
Oct 30 20:11:21.945304 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.945297    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2" event={"ID":"aecc895e-a6d4-4fc6-8e55-92b83f17fcde","Type":"ContainerDied","Data":"4eb9c8492ccab967b51c9ca40f381befe4bf916a2cf1d57ec6ee9dbc0f0ecc8b"}
Oct 30 20:11:21.956457 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.956428    5408 scope.go:122] "RemoveContainer" containerID="8ab54f0292ee3bf38e728dd38f96b66de6b552caf8974b2e129cfa61c7ff08a5"
Oct 30 20:11:21.956887 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:11:21.956861    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab54f0292ee3bf38e728dd38f96b66de6b552caf8974b2e129cfa61c7ff08a5\": container with ID starting with 8ab54f0292ee3bf38e728dd38f96b66de6b552caf8974b2e129cfa61c7ff08a5 not found: ID does not exist" containerID="8ab54f0292ee3bf38e728dd38f96b66de6b552caf8974b2e129cfa61c7ff08a5"
Oct 30 20:11:21.957009 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.956895    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab54f0292ee3bf38e728dd38f96b66de6b552caf8974b2e129cfa61c7ff08a5"} err="failed to get container status \"8ab54f0292ee3bf38e728dd38f96b66de6b552caf8974b2e129cfa61c7ff08a5\": rpc error: code = NotFound desc = could not find container \"8ab54f0292ee3bf38e728dd38f96b66de6b552caf8974b2e129cfa61c7ff08a5\": container with ID starting with 8ab54f0292ee3bf38e728dd38f96b66de6b552caf8974b2e129cfa61c7ff08a5 not found: ID does not exist"
Oct 30 20:11:21.957009 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.956915    5408 scope.go:122] "RemoveContainer" containerID="ad580d4ee05705771ef6145063227f407db1caa8dff11e02721c78abf10fff09"
Oct 30 20:11:21.972354 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.972333    5408 scope.go:122] "RemoveContainer" containerID="ad580d4ee05705771ef6145063227f407db1caa8dff11e02721c78abf10fff09"
Oct 30 20:11:21.972758 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:11:21.972728    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad580d4ee05705771ef6145063227f407db1caa8dff11e02721c78abf10fff09\": container with ID starting with ad580d4ee05705771ef6145063227f407db1caa8dff11e02721c78abf10fff09 not found: ID does not exist" containerID="ad580d4ee05705771ef6145063227f407db1caa8dff11e02721c78abf10fff09"
Oct 30 20:11:21.972875 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:21.972797    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad580d4ee05705771ef6145063227f407db1caa8dff11e02721c78abf10fff09"} err="failed to get container status \"ad580d4ee05705771ef6145063227f407db1caa8dff11e02721c78abf10fff09\": rpc error: code = NotFound desc = could not find container \"ad580d4ee05705771ef6145063227f407db1caa8dff11e02721c78abf10fff09\": container with ID starting with ad580d4ee05705771ef6145063227f407db1caa8dff11e02721c78abf10fff09 not found: ID does not exist"
Oct 30 20:11:22.076084 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.075865    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmx8l\" (UniqueName: \"kubernetes.io/projected/aecc895e-a6d4-4fc6-8e55-92b83f17fcde-kube-api-access-vmx8l\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:22.093903 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.093808    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p"]
Oct 30 20:11:22.097880 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.097841    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-db02a7-8585ffff59-gbx9p"]
Oct 30 20:11:22.134071 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.123153    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"]
Oct 30 20:11:22.449246 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.449146    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2"]
Oct 30 20:11:22.464096 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.464066    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-db02a7-cb4c5d8d6-6m9x2"]
Oct 30 20:11:22.568775 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.568738    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8"]
Oct 30 20:11:22.573538 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.573508    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 20:11:22.618121 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.618087    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl"]
Oct 30 20:11:22.640850 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.640723    5408 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 20:11:22.667222 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.667181    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9" podUID="4ddd1055-47a1-4c05-bd03-ff3c493f32a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 20:11:22.719127 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.718922    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-podinfo\") pod \"88493f4b-7ec6-4f72-92df-03fb99d9c612\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") "
Oct 30 20:11:22.719127 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.719012    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-workload-certs\") pod \"88493f4b-7ec6-4f72-92df-03fb99d9c612\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") "
Oct 30 20:11:22.719127 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.719093    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88493f4b-7ec6-4f72-92df-03fb99d9c612-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88493f4b-7ec6-4f72-92df-03fb99d9c612-istiod-ca-cert\") pod \"88493f4b-7ec6-4f72-92df-03fb99d9c612\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") "
Oct 30 20:11:22.719127 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.719124    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88493f4b-7ec6-4f72-92df-03fb99d9c612-kube-api-access-z6qgw\" (UniqueName: \"kubernetes.io/projected/88493f4b-7ec6-4f72-92df-03fb99d9c612-kube-api-access-z6qgw\") pod \"88493f4b-7ec6-4f72-92df-03fb99d9c612\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") "
Oct 30 20:11:22.719494 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.719182    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-token\" (UniqueName: \"kubernetes.io/projected/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-token\") pod \"88493f4b-7ec6-4f72-92df-03fb99d9c612\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") "
Oct 30 20:11:22.719494 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.719254    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-credential-socket\") pod \"88493f4b-7ec6-4f72-92df-03fb99d9c612\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") "
Oct 30 20:11:22.719494 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.719298    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-data\") pod \"88493f4b-7ec6-4f72-92df-03fb99d9c612\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") "
Oct 30 20:11:22.719494 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.719429    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-envoy\") pod \"88493f4b-7ec6-4f72-92df-03fb99d9c612\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") "
Oct 30 20:11:22.719707 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.719619    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-workload-socket\") pod \"88493f4b-7ec6-4f72-92df-03fb99d9c612\" (UID: \"88493f4b-7ec6-4f72-92df-03fb99d9c612\") "
Oct 30 20:11:22.740770 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.740724    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-credential-socket" pod "88493f4b-7ec6-4f72-92df-03fb99d9c612" (UID: "88493f4b-7ec6-4f72-92df-03fb99d9c612"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:22.741332 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.741271    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-workload-certs" pod "88493f4b-7ec6-4f72-92df-03fb99d9c612" (UID: "88493f4b-7ec6-4f72-92df-03fb99d9c612"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:22.741940 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.741917    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88493f4b-7ec6-4f72-92df-03fb99d9c612-istiod-ca-cert" pod "88493f4b-7ec6-4f72-92df-03fb99d9c612" (UID: "88493f4b-7ec6-4f72-92df-03fb99d9c612"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:22.742465 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.742431    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-data" pod "88493f4b-7ec6-4f72-92df-03fb99d9c612" (UID: "88493f4b-7ec6-4f72-92df-03fb99d9c612"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:22.742694 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.742659    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-workload-socket" pod "88493f4b-7ec6-4f72-92df-03fb99d9c612" (UID: "88493f4b-7ec6-4f72-92df-03fb99d9c612"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:22.751593 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.751536    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-token" pod "88493f4b-7ec6-4f72-92df-03fb99d9c612" (UID: "88493f4b-7ec6-4f72-92df-03fb99d9c612"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:22.762340 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.762304    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88493f4b-7ec6-4f72-92df-03fb99d9c612-kube-api-access-z6qgw" pod "88493f4b-7ec6-4f72-92df-03fb99d9c612" (UID: "88493f4b-7ec6-4f72-92df-03fb99d9c612"). InnerVolumeSpecName "kube-api-access-z6qgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:22.763438 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.763409    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-envoy" pod "88493f4b-7ec6-4f72-92df-03fb99d9c612" (UID: "88493f4b-7ec6-4f72-92df-03fb99d9c612"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:22.764203 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.764167    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-podinfo" pod "88493f4b-7ec6-4f72-92df-03fb99d9c612" (UID: "88493f4b-7ec6-4f72-92df-03fb99d9c612"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:11:22.842103 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.842011    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:22.842103 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.842106    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:22.842499 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.842121    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:22.842499 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.842133    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:22.842499 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.842147    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:22.842499 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.842162    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88493f4b-7ec6-4f72-92df-03fb99d9c612-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:22.842499 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.842174    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88493f4b-7ec6-4f72-92df-03fb99d9c612-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:22.842499 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.842188    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6qgw\" (UniqueName: \"kubernetes.io/projected/88493f4b-7ec6-4f72-92df-03fb99d9c612-kube-api-access-z6qgw\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:22.842499 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.842202    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88493f4b-7ec6-4f72-92df-03fb99d9c612-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:22.960511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.960477    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"
Oct 30 20:11:22.960671 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.960478    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg" event={"ID":"88493f4b-7ec6-4f72-92df-03fb99d9c612","Type":"ContainerDied","Data":"d8bf9b472b8bc63057d3aa6a7dded0450b25490abb361cd3b25a48a7347570ec"}
Oct 30 20:11:22.960671 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:22.960623    5408 scope.go:122] "RemoveContainer" containerID="740036cd4ff7c7c2f138fd5a15eea5bf22eb69a7c272e192f8fec8ff9e72da9f"
Oct 30 20:11:23.058172 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:23.039309    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"]
Oct 30 20:11:23.110869 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:23.110828    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-4zppg"]
Oct 30 20:11:23.546334 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:23.546296    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651ba59f-3f1f-479d-805e-30a17661c602" path="/var/lib/kubelet/pods/651ba59f-3f1f-479d-805e-30a17661c602/volumes"
Oct 30 20:11:23.546901 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:23.546884    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88493f4b-7ec6-4f72-92df-03fb99d9c612" path="/var/lib/kubelet/pods/88493f4b-7ec6-4f72-92df-03fb99d9c612/volumes"
Oct 30 20:11:23.547350 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:23.547336    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aecc895e-a6d4-4fc6-8e55-92b83f17fcde" path="/var/lib/kubelet/pods/aecc895e-a6d4-4fc6-8e55-92b83f17fcde/volumes"
Oct 30 20:11:23.630420 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:23.629592    5408 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 20:11:23.630734 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:23.630699    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9" podUID="4ddd1055-47a1-4c05-bd03-ff3c493f32a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 20:11:24.161565 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:24.161445    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"]
Oct 30 20:11:24.623951 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:24.623914    5408 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 20:11:24.624341 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:24.624312    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9" podUID="4ddd1055-47a1-4c05-bd03-ff3c493f32a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 20:11:25.134947 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.134876    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29" podUID="3558d905-d0a3-4b3f-8c5c-dac3353f4b1c" containerName="main" containerID="cri-o://15a59837dbd6ce9f48f96f7adfbebd4222060471eaf4a231e4f9e144ada8ea80" gracePeriod=45
Oct 30 20:11:25.163251 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.163205    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29" podUID="3558d905-d0a3-4b3f-8c5c-dac3353f4b1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.156:8000/ready\": dial tcp 10.134.0.156:8000: connect: connection refused"
Oct 30 20:11:25.529962 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.529934    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:11:25.621214 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.621182    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-home\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-home\") pod \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") "
Oct 30 20:11:25.621431 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.621245    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-kube-api-access-hz5mt\" (UniqueName: \"kubernetes.io/projected/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-kube-api-access-hz5mt\") pod \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") "
Oct 30 20:11:25.621431 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.621324    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-model-cache\") pod \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") "
Oct 30 20:11:25.621545 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.621468    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-tmp-dir\") pod \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") "
Oct 30 20:11:25.621545 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.621506    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-dshm\") pod \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") "
Oct 30 20:11:25.621545 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.621527    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-tls-certs\" (UniqueName: \"kubernetes.io/secret/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-tls-certs\") pod \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\" (UID: \"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c\") "
Oct 30 20:11:25.621694 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.621513    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-home" pod "3558d905-d0a3-4b3f-8c5c-dac3353f4b1c" (UID: "3558d905-d0a3-4b3f-8c5c-dac3353f4b1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:25.621694 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.621589    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-model-cache" pod "3558d905-d0a3-4b3f-8c5c-dac3353f4b1c" (UID: "3558d905-d0a3-4b3f-8c5c-dac3353f4b1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:25.621804 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.621713    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-tmp-dir" pod "3558d905-d0a3-4b3f-8c5c-dac3353f4b1c" (UID: "3558d905-d0a3-4b3f-8c5c-dac3353f4b1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:25.622626 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.622603    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:25.622626 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.622628    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:25.622810 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.622642    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:25.623906 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.623881    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-dshm" pod "3558d905-d0a3-4b3f-8c5c-dac3353f4b1c" (UID: "3558d905-d0a3-4b3f-8c5c-dac3353f4b1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:25.624167 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.624145    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-kube-api-access-hz5mt" pod "3558d905-d0a3-4b3f-8c5c-dac3353f4b1c" (UID: "3558d905-d0a3-4b3f-8c5c-dac3353f4b1c"). InnerVolumeSpecName "kube-api-access-hz5mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:25.624167 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.624156    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-tls-certs" pod "3558d905-d0a3-4b3f-8c5c-dac3353f4b1c" (UID: "3558d905-d0a3-4b3f-8c5c-dac3353f4b1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:25.629545 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.629518    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:25.629958 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.629937    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:25.630659 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.630640    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:25.724971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.724939    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:25.724971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.724975    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:25.725248 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.724991    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz5mt\" (UniqueName: \"kubernetes.io/projected/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c-kube-api-access-hz5mt\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:25.994687 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.994650    5408 generic.go:356] "Generic (PLEG): container finished" podID="3558d905-d0a3-4b3f-8c5c-dac3353f4b1c" containerID="15a59837dbd6ce9f48f96f7adfbebd4222060471eaf4a231e4f9e144ada8ea80" exitCode=0
Oct 30 20:11:25.994903 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.994705    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"
Oct 30 20:11:25.994903 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.994762    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29" event={"ID":"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c","Type":"ContainerDied","Data":"15a59837dbd6ce9f48f96f7adfbebd4222060471eaf4a231e4f9e144ada8ea80"}
Oct 30 20:11:25.994903 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.994829    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29" event={"ID":"3558d905-d0a3-4b3f-8c5c-dac3353f4b1c","Type":"ContainerDied","Data":"cff2a512f0d4f1505066790ab08a317d94bbc4d4409f5fad332f52581287a6c0"}
Oct 30 20:11:25.994903 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:25.994857    5408 scope.go:122] "RemoveContainer" containerID="15a59837dbd6ce9f48f96f7adfbebd4222060471eaf4a231e4f9e144ada8ea80"
Oct 30 20:11:26.009255 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:26.009235    5408 scope.go:122] "RemoveContainer" containerID="15a59837dbd6ce9f48f96f7adfbebd4222060471eaf4a231e4f9e144ada8ea80"
Oct 30 20:11:26.009634 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:11:26.009614    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a59837dbd6ce9f48f96f7adfbebd4222060471eaf4a231e4f9e144ada8ea80\": container with ID starting with 15a59837dbd6ce9f48f96f7adfbebd4222060471eaf4a231e4f9e144ada8ea80 not found: ID does not exist" containerID="15a59837dbd6ce9f48f96f7adfbebd4222060471eaf4a231e4f9e144ada8ea80"
Oct 30 20:11:26.009691 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:26.009645    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a59837dbd6ce9f48f96f7adfbebd4222060471eaf4a231e4f9e144ada8ea80"} err="failed to get container status \"15a59837dbd6ce9f48f96f7adfbebd4222060471eaf4a231e4f9e144ada8ea80\": rpc error: code = NotFound desc = could not find container \"15a59837dbd6ce9f48f96f7adfbebd4222060471eaf4a231e4f9e144ada8ea80\": container with ID starting with 15a59837dbd6ce9f48f96f7adfbebd4222060471eaf4a231e4f9e144ada8ea80 not found: ID does not exist"
Oct 30 20:11:26.027160 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:26.027126    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"]
Oct 30 20:11:26.030436 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:26.030409    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3a4f02/test-model-3a4f02-kserve-7d45cc4755-nqd29"]
Oct 30 20:11:27.072487 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.072397    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z" podUID="723bb78d-b254-4cea-bf5e-6e5609a8a699" containerName="main" containerID="cri-o://e0fc6e8036e3c34dd9bd0f119e2f9afbc207e34650d206caaf4b4b9cf5e32f2e" gracePeriod=45
Oct 30 20:11:27.439054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.437436    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:11:27.486071 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.486021    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/723bb78d-b254-4cea-bf5e-6e5609a8a699-tls-certs\" (UniqueName: \"kubernetes.io/secret/723bb78d-b254-4cea-bf5e-6e5609a8a699-tls-certs\") pod \"723bb78d-b254-4cea-bf5e-6e5609a8a699\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") "
Oct 30 20:11:27.486277 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.486209    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723bb78d-b254-4cea-bf5e-6e5609a8a699-kube-api-access-rwtgd\" (UniqueName: \"kubernetes.io/projected/723bb78d-b254-4cea-bf5e-6e5609a8a699-kube-api-access-rwtgd\") pod \"723bb78d-b254-4cea-bf5e-6e5609a8a699\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") "
Oct 30 20:11:27.486277 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.486253    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-home\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-home\") pod \"723bb78d-b254-4cea-bf5e-6e5609a8a699\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") "
Oct 30 20:11:27.486396 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.486364    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-dshm\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-dshm\") pod \"723bb78d-b254-4cea-bf5e-6e5609a8a699\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") "
Oct 30 20:11:27.486537 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.486521    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-model-cache\") pod \"723bb78d-b254-4cea-bf5e-6e5609a8a699\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") "
Oct 30 20:11:27.486606 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.486568    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-tmp-dir\") pod \"723bb78d-b254-4cea-bf5e-6e5609a8a699\" (UID: \"723bb78d-b254-4cea-bf5e-6e5609a8a699\") "
Oct 30 20:11:27.487680 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.487651    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-tmp-dir" pod "723bb78d-b254-4cea-bf5e-6e5609a8a699" (UID: "723bb78d-b254-4cea-bf5e-6e5609a8a699"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:27.487855 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.487837    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-home" pod "723bb78d-b254-4cea-bf5e-6e5609a8a699" (UID: "723bb78d-b254-4cea-bf5e-6e5609a8a699"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:27.488348 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.488320    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-model-cache" pod "723bb78d-b254-4cea-bf5e-6e5609a8a699" (UID: "723bb78d-b254-4cea-bf5e-6e5609a8a699"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:27.490319 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.489598    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/723bb78d-b254-4cea-bf5e-6e5609a8a699-tls-certs" pod "723bb78d-b254-4cea-bf5e-6e5609a8a699" (UID: "723bb78d-b254-4cea-bf5e-6e5609a8a699"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:27.491727 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.491668    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-dshm" pod "723bb78d-b254-4cea-bf5e-6e5609a8a699" (UID: "723bb78d-b254-4cea-bf5e-6e5609a8a699"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:27.496709 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.496672    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723bb78d-b254-4cea-bf5e-6e5609a8a699-kube-api-access-rwtgd" pod "723bb78d-b254-4cea-bf5e-6e5609a8a699" (UID: "723bb78d-b254-4cea-bf5e-6e5609a8a699"). InnerVolumeSpecName "kube-api-access-rwtgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:27.533621 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.533595    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3558d905-d0a3-4b3f-8c5c-dac3353f4b1c" path="/var/lib/kubelet/pods/3558d905-d0a3-4b3f-8c5c-dac3353f4b1c/volumes"
Oct 30 20:11:27.569867 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.569835    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8" podUID="415f31e1-c808-4fb0-bb8c-33f928248892" containerName="payload-processing" containerID="cri-o://ab9651b4b3424af3dbdd4181cecbad74df4bcca41bdbfef7686f0e4f08a53ec9" gracePeriod=25
Oct 30 20:11:27.589724 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.589664    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/723bb78d-b254-4cea-bf5e-6e5609a8a699-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:27.589724 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.589695    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwtgd\" (UniqueName: \"kubernetes.io/projected/723bb78d-b254-4cea-bf5e-6e5609a8a699-kube-api-access-rwtgd\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:27.589724 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.589712    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:27.589890 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.589727    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:27.589890 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.589742    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:27.589890 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.589756    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/723bb78d-b254-4cea-bf5e-6e5609a8a699-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:27.653843 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.653812    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl" podUID="b599d3f2-4e57-4072-b8f9-02e0d8422678" containerName="payload-pre-processing" containerID="cri-o://74ed8dea250356ad06cfa8ee944cd5f882488476a72f84f503930ce47b79fbb0" gracePeriod=25
Oct 30 20:11:27.906236 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.906211    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8"
Oct 30 20:11:27.975853 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.975825    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl"
Oct 30 20:11:27.996099 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.995986    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/415f31e1-c808-4fb0-bb8c-33f928248892-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415f31e1-c808-4fb0-bb8c-33f928248892-plugins-config-volume\") pod \"415f31e1-c808-4fb0-bb8c-33f928248892\" (UID: \"415f31e1-c808-4fb0-bb8c-33f928248892\") "
Oct 30 20:11:27.996262 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.996108    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415f31e1-c808-4fb0-bb8c-33f928248892-kube-api-access-tb72k\" (UniqueName: \"kubernetes.io/projected/415f31e1-c808-4fb0-bb8c-33f928248892-kube-api-access-tb72k\") pod \"415f31e1-c808-4fb0-bb8c-33f928248892\" (UID: \"415f31e1-c808-4fb0-bb8c-33f928248892\") "
Oct 30 20:11:27.996503 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.996428    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415f31e1-c808-4fb0-bb8c-33f928248892-plugins-config-volume" pod "415f31e1-c808-4fb0-bb8c-33f928248892" (UID: "415f31e1-c808-4fb0-bb8c-33f928248892"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:27.997180 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.997156    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415f31e1-c808-4fb0-bb8c-33f928248892-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:27.999299 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:27.999271    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415f31e1-c808-4fb0-bb8c-33f928248892-kube-api-access-tb72k" pod "415f31e1-c808-4fb0-bb8c-33f928248892" (UID: "415f31e1-c808-4fb0-bb8c-33f928248892"). InnerVolumeSpecName "kube-api-access-tb72k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:28.021421 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.021387    5408 generic.go:356] "Generic (PLEG): container finished" podID="723bb78d-b254-4cea-bf5e-6e5609a8a699" containerID="e0fc6e8036e3c34dd9bd0f119e2f9afbc207e34650d206caaf4b4b9cf5e32f2e" exitCode=0
Oct 30 20:11:28.021421 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.021439    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"
Oct 30 20:11:28.021676 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.021465    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z" event={"ID":"723bb78d-b254-4cea-bf5e-6e5609a8a699","Type":"ContainerDied","Data":"e0fc6e8036e3c34dd9bd0f119e2f9afbc207e34650d206caaf4b4b9cf5e32f2e"}
Oct 30 20:11:28.021676 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.021532    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z" event={"ID":"723bb78d-b254-4cea-bf5e-6e5609a8a699","Type":"ContainerDied","Data":"fc0cedb4340b2717f284bd697862e138bd2c730d8e893b9a513fd4280c702ba7"}
Oct 30 20:11:28.021676 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.021562    5408 scope.go:122] "RemoveContainer" containerID="e0fc6e8036e3c34dd9bd0f119e2f9afbc207e34650d206caaf4b4b9cf5e32f2e"
Oct 30 20:11:28.028126 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.024418    5408 generic.go:356] "Generic (PLEG): container finished" podID="415f31e1-c808-4fb0-bb8c-33f928248892" containerID="ab9651b4b3424af3dbdd4181cecbad74df4bcca41bdbfef7686f0e4f08a53ec9" exitCode=0
Oct 30 20:11:28.028126 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.024532    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8" event={"ID":"415f31e1-c808-4fb0-bb8c-33f928248892","Type":"ContainerDied","Data":"ab9651b4b3424af3dbdd4181cecbad74df4bcca41bdbfef7686f0e4f08a53ec9"}
Oct 30 20:11:28.028126 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.024576    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8" event={"ID":"415f31e1-c808-4fb0-bb8c-33f928248892","Type":"ContainerDied","Data":"6b76c53434dda726da0caaceda5b54294f9910b6c20b11735d69f15c2337256a"}
Oct 30 20:11:28.028126 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.024656    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8"
Oct 30 20:11:28.030244 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.030219    5408 generic.go:356] "Generic (PLEG): container finished" podID="b599d3f2-4e57-4072-b8f9-02e0d8422678" containerID="74ed8dea250356ad06cfa8ee944cd5f882488476a72f84f503930ce47b79fbb0" exitCode=0
Oct 30 20:11:28.030366 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.030268    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl"
Oct 30 20:11:28.030366 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.030328    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl" event={"ID":"b599d3f2-4e57-4072-b8f9-02e0d8422678","Type":"ContainerDied","Data":"74ed8dea250356ad06cfa8ee944cd5f882488476a72f84f503930ce47b79fbb0"}
Oct 30 20:11:28.030491 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.030397    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl" event={"ID":"b599d3f2-4e57-4072-b8f9-02e0d8422678","Type":"ContainerDied","Data":"2e6547f7040bf7f210e0ad6e47d46d6fcbc459eccb2423de1c974061a8e77b0a"}
Oct 30 20:11:28.040259 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.040231    5408 scope.go:122] "RemoveContainer" containerID="e0fc6e8036e3c34dd9bd0f119e2f9afbc207e34650d206caaf4b4b9cf5e32f2e"
Oct 30 20:11:28.040701 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:11:28.040675    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0fc6e8036e3c34dd9bd0f119e2f9afbc207e34650d206caaf4b4b9cf5e32f2e\": container with ID starting with e0fc6e8036e3c34dd9bd0f119e2f9afbc207e34650d206caaf4b4b9cf5e32f2e not found: ID does not exist" containerID="e0fc6e8036e3c34dd9bd0f119e2f9afbc207e34650d206caaf4b4b9cf5e32f2e"
Oct 30 20:11:28.040802 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.040713    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0fc6e8036e3c34dd9bd0f119e2f9afbc207e34650d206caaf4b4b9cf5e32f2e"} err="failed to get container status \"e0fc6e8036e3c34dd9bd0f119e2f9afbc207e34650d206caaf4b4b9cf5e32f2e\": rpc error: code = NotFound desc = could not find container \"e0fc6e8036e3c34dd9bd0f119e2f9afbc207e34650d206caaf4b4b9cf5e32f2e\": container with ID starting with e0fc6e8036e3c34dd9bd0f119e2f9afbc207e34650d206caaf4b4b9cf5e32f2e not found: ID does not exist"
Oct 30 20:11:28.040802 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.040743    5408 scope.go:122] "RemoveContainer" containerID="ab9651b4b3424af3dbdd4181cecbad74df4bcca41bdbfef7686f0e4f08a53ec9"
Oct 30 20:11:28.060776 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.060757    5408 scope.go:122] "RemoveContainer" containerID="ab9651b4b3424af3dbdd4181cecbad74df4bcca41bdbfef7686f0e4f08a53ec9"
Oct 30 20:11:28.061063 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:11:28.061042    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9651b4b3424af3dbdd4181cecbad74df4bcca41bdbfef7686f0e4f08a53ec9\": container with ID starting with ab9651b4b3424af3dbdd4181cecbad74df4bcca41bdbfef7686f0e4f08a53ec9 not found: ID does not exist" containerID="ab9651b4b3424af3dbdd4181cecbad74df4bcca41bdbfef7686f0e4f08a53ec9"
Oct 30 20:11:28.061159 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.061076    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9651b4b3424af3dbdd4181cecbad74df4bcca41bdbfef7686f0e4f08a53ec9"} err="failed to get container status \"ab9651b4b3424af3dbdd4181cecbad74df4bcca41bdbfef7686f0e4f08a53ec9\": rpc error: code = NotFound desc = could not find container \"ab9651b4b3424af3dbdd4181cecbad74df4bcca41bdbfef7686f0e4f08a53ec9\": container with ID starting with ab9651b4b3424af3dbdd4181cecbad74df4bcca41bdbfef7686f0e4f08a53ec9 not found: ID does not exist"
Oct 30 20:11:28.061159 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.061100    5408 scope.go:122] "RemoveContainer" containerID="74ed8dea250356ad06cfa8ee944cd5f882488476a72f84f503930ce47b79fbb0"
Oct 30 20:11:28.069911 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.069885    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"]
Oct 30 20:11:28.073948 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.073929    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z"]
Oct 30 20:11:28.074500 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.074426    5408 scope.go:122] "RemoveContainer" containerID="74ed8dea250356ad06cfa8ee944cd5f882488476a72f84f503930ce47b79fbb0"
Oct 30 20:11:28.074706 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:11:28.074689    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74ed8dea250356ad06cfa8ee944cd5f882488476a72f84f503930ce47b79fbb0\": container with ID starting with 74ed8dea250356ad06cfa8ee944cd5f882488476a72f84f503930ce47b79fbb0 not found: ID does not exist" containerID="74ed8dea250356ad06cfa8ee944cd5f882488476a72f84f503930ce47b79fbb0"
Oct 30 20:11:28.074751 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.074713    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74ed8dea250356ad06cfa8ee944cd5f882488476a72f84f503930ce47b79fbb0"} err="failed to get container status \"74ed8dea250356ad06cfa8ee944cd5f882488476a72f84f503930ce47b79fbb0\": rpc error: code = NotFound desc = could not find container \"74ed8dea250356ad06cfa8ee944cd5f882488476a72f84f503930ce47b79fbb0\": container with ID starting with 74ed8dea250356ad06cfa8ee944cd5f882488476a72f84f503930ce47b79fbb0 not found: ID does not exist"
Oct 30 20:11:28.086616 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.086594    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8"]
Oct 30 20:11:28.097429 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.097408    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a4f02-6c57677b4-whrn8"]
Oct 30 20:11:28.098059 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.098010    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b599d3f2-4e57-4072-b8f9-02e0d8422678-kube-api-access-4246p\" (UniqueName: \"kubernetes.io/projected/b599d3f2-4e57-4072-b8f9-02e0d8422678-kube-api-access-4246p\") pod \"b599d3f2-4e57-4072-b8f9-02e0d8422678\" (UID: \"b599d3f2-4e57-4072-b8f9-02e0d8422678\") "
Oct 30 20:11:28.098337 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.098316    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b599d3f2-4e57-4072-b8f9-02e0d8422678-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b599d3f2-4e57-4072-b8f9-02e0d8422678-plugins-config-volume\") pod \"b599d3f2-4e57-4072-b8f9-02e0d8422678\" (UID: \"b599d3f2-4e57-4072-b8f9-02e0d8422678\") "
Oct 30 20:11:28.098685 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.098656    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b599d3f2-4e57-4072-b8f9-02e0d8422678-plugins-config-volume" pod "b599d3f2-4e57-4072-b8f9-02e0d8422678" (UID: "b599d3f2-4e57-4072-b8f9-02e0d8422678"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:28.099084 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.099066    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b599d3f2-4e57-4072-b8f9-02e0d8422678-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:28.099136 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.099086    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb72k\" (UniqueName: \"kubernetes.io/projected/415f31e1-c808-4fb0-bb8c-33f928248892-kube-api-access-tb72k\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:28.100241 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.100214    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b599d3f2-4e57-4072-b8f9-02e0d8422678-kube-api-access-4246p" pod "b599d3f2-4e57-4072-b8f9-02e0d8422678" (UID: "b599d3f2-4e57-4072-b8f9-02e0d8422678"). InnerVolumeSpecName "kube-api-access-4246p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:28.201197 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.201166    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4246p\" (UniqueName: \"kubernetes.io/projected/b599d3f2-4e57-4072-b8f9-02e0d8422678-kube-api-access-4246p\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:28.358522 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.358478    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl"]
Oct 30 20:11:28.364600 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.364571    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3a4f02-cb98877f-zx8vl"]
Oct 30 20:11:28.385812 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.385738    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-82885f/test-model-82885f-kserve-f5cf59b77-f845z" podUID="723bb78d-b254-4cea-bf5e-6e5609a8a699" containerName="main" probeResult="failure" output="Get \"https://10.134.0.157:8000/ready\": context deadline exceeded"
Oct 30 20:11:28.844771 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.844716    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"]
Oct 30 20:11:28.845704 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:28.845668    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9" podUID="4ddd1055-47a1-4c05-bd03-ff3c493f32a7" containerName="istio-proxy" containerID="cri-o://7dc0c20e82b2261a27efba3bb9fabb0e26e4254dbb3737f63de9b0c47c03ae9e" gracePeriod=30
Oct 30 20:11:29.072836 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.069715    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb" podUID="9b56dd57-5a11-4834-a128-5aa9e2385157" containerName="main" containerID="cri-o://9961129148dfea086f748fdca856825bc06c50d422b7bf2e7be3d35af151160f" gracePeriod=45
Oct 30 20:11:29.534424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.534383    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="415f31e1-c808-4fb0-bb8c-33f928248892" path="/var/lib/kubelet/pods/415f31e1-c808-4fb0-bb8c-33f928248892/volumes"
Oct 30 20:11:29.534994 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.534975    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="723bb78d-b254-4cea-bf5e-6e5609a8a699" path="/var/lib/kubelet/pods/723bb78d-b254-4cea-bf5e-6e5609a8a699/volumes"
Oct 30 20:11:29.535416 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.535399    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b599d3f2-4e57-4072-b8f9-02e0d8422678" path="/var/lib/kubelet/pods/b599d3f2-4e57-4072-b8f9-02e0d8422678/volumes"
Oct 30 20:11:29.540719 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.540697    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:11:29.636513 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.636480    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-model-cache\") pod \"9b56dd57-5a11-4834-a128-5aa9e2385157\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") "
Oct 30 20:11:29.638153 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.636620    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-tmp-dir\") pod \"9b56dd57-5a11-4834-a128-5aa9e2385157\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") "
Oct 30 20:11:29.638153 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.636784    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-model-cache" pod "9b56dd57-5a11-4834-a128-5aa9e2385157" (UID: "9b56dd57-5a11-4834-a128-5aa9e2385157"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:29.638153 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.637068    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-tmp-dir" pod "9b56dd57-5a11-4834-a128-5aa9e2385157" (UID: "9b56dd57-5a11-4834-a128-5aa9e2385157"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:29.642148 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.642092    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-home\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-home\") pod \"9b56dd57-5a11-4834-a128-5aa9e2385157\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") "
Oct 30 20:11:29.642301 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.642249    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b56dd57-5a11-4834-a128-5aa9e2385157-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b56dd57-5a11-4834-a128-5aa9e2385157-tls-certs\") pod \"9b56dd57-5a11-4834-a128-5aa9e2385157\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") "
Oct 30 20:11:29.642469 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.642448    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-home" pod "9b56dd57-5a11-4834-a128-5aa9e2385157" (UID: "9b56dd57-5a11-4834-a128-5aa9e2385157"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:29.642545 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.642496    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b56dd57-5a11-4834-a128-5aa9e2385157-kube-api-access-npbmf\" (UniqueName: \"kubernetes.io/projected/9b56dd57-5a11-4834-a128-5aa9e2385157-kube-api-access-npbmf\") pod \"9b56dd57-5a11-4834-a128-5aa9e2385157\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") "
Oct 30 20:11:29.642603 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.642555    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-dshm\") pod \"9b56dd57-5a11-4834-a128-5aa9e2385157\" (UID: \"9b56dd57-5a11-4834-a128-5aa9e2385157\") "
Oct 30 20:11:29.643257 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.643233    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:29.643351 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.643260    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:29.643351 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.643274    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:29.645411 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.645388    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-dshm" pod "9b56dd57-5a11-4834-a128-5aa9e2385157" (UID: "9b56dd57-5a11-4834-a128-5aa9e2385157"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:29.645519 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.645448    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b56dd57-5a11-4834-a128-5aa9e2385157-kube-api-access-npbmf" pod "9b56dd57-5a11-4834-a128-5aa9e2385157" (UID: "9b56dd57-5a11-4834-a128-5aa9e2385157"). InnerVolumeSpecName "kube-api-access-npbmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:29.646425 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.646402    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b56dd57-5a11-4834-a128-5aa9e2385157-tls-certs" pod "9b56dd57-5a11-4834-a128-5aa9e2385157" (UID: "9b56dd57-5a11-4834-a128-5aa9e2385157"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:29.744855 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.744817    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npbmf\" (UniqueName: \"kubernetes.io/projected/9b56dd57-5a11-4834-a128-5aa9e2385157-kube-api-access-npbmf\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:29.744855 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.744859    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b56dd57-5a11-4834-a128-5aa9e2385157-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:29.745068 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:29.744875    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b56dd57-5a11-4834-a128-5aa9e2385157-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:30.108865 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:30.108827    5408 generic.go:356] "Generic (PLEG): container finished" podID="9b56dd57-5a11-4834-a128-5aa9e2385157" containerID="9961129148dfea086f748fdca856825bc06c50d422b7bf2e7be3d35af151160f" exitCode=0
Oct 30 20:11:30.109115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:30.108881    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"
Oct 30 20:11:30.109115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:30.108926    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb" event={"ID":"9b56dd57-5a11-4834-a128-5aa9e2385157","Type":"ContainerDied","Data":"9961129148dfea086f748fdca856825bc06c50d422b7bf2e7be3d35af151160f"}
Oct 30 20:11:30.109115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:30.108970    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb" event={"ID":"9b56dd57-5a11-4834-a128-5aa9e2385157","Type":"ContainerDied","Data":"6ce0e33ec38118cb7ca7a2e1627d6db8a73fdbed094251e7be406b9c09ffca33"}
Oct 30 20:11:30.109115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:30.108987    5408 scope.go:122] "RemoveContainer" containerID="9961129148dfea086f748fdca856825bc06c50d422b7bf2e7be3d35af151160f"
Oct 30 20:11:30.122917 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:30.122890    5408 scope.go:122] "RemoveContainer" containerID="9961129148dfea086f748fdca856825bc06c50d422b7bf2e7be3d35af151160f"
Oct 30 20:11:30.123299 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:11:30.123279    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9961129148dfea086f748fdca856825bc06c50d422b7bf2e7be3d35af151160f\": container with ID starting with 9961129148dfea086f748fdca856825bc06c50d422b7bf2e7be3d35af151160f not found: ID does not exist" containerID="9961129148dfea086f748fdca856825bc06c50d422b7bf2e7be3d35af151160f"
Oct 30 20:11:30.123355 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:30.123310    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9961129148dfea086f748fdca856825bc06c50d422b7bf2e7be3d35af151160f"} err="failed to get container status \"9961129148dfea086f748fdca856825bc06c50d422b7bf2e7be3d35af151160f\": rpc error: code = NotFound desc = could not find container \"9961129148dfea086f748fdca856825bc06c50d422b7bf2e7be3d35af151160f\": container with ID starting with 9961129148dfea086f748fdca856825bc06c50d422b7bf2e7be3d35af151160f not found: ID does not exist"
Oct 30 20:11:30.140782 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:30.140757    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"]
Oct 30 20:11:30.146431 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:30.146411    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a7bb6c/test-model-a7bb6c-kserve-7b6df65d9-5wcfb"]
Oct 30 20:11:30.535733 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:30.535699    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"]
Oct 30 20:11:31.542366 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:31.542328    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b56dd57-5a11-4834-a128-5aa9e2385157" path="/var/lib/kubelet/pods/9b56dd57-5a11-4834-a128-5aa9e2385157/volumes"
Oct 30 20:11:34.211705 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.211678    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:34.218324 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.218298    5408 generic.go:356] "Generic (PLEG): container finished" podID="4ddd1055-47a1-4c05-bd03-ff3c493f32a7" containerID="7dc0c20e82b2261a27efba3bb9fabb0e26e4254dbb3737f63de9b0c47c03ae9e" exitCode=0
Oct 30 20:11:34.218445 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.218351    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"
Oct 30 20:11:34.218445 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.218425    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9" event={"ID":"4ddd1055-47a1-4c05-bd03-ff3c493f32a7","Type":"ContainerDied","Data":"7dc0c20e82b2261a27efba3bb9fabb0e26e4254dbb3737f63de9b0c47c03ae9e"}
Oct 30 20:11:34.218537 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.218475    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9" event={"ID":"4ddd1055-47a1-4c05-bd03-ff3c493f32a7","Type":"ContainerDied","Data":"eebd0cdfed94002524abb96e1d05a58e5ed4fd9747ac2a84a90545e3a2e40c1d"}
Oct 30 20:11:34.218537 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.218498    5408 scope.go:122] "RemoveContainer" containerID="7dc0c20e82b2261a27efba3bb9fabb0e26e4254dbb3737f63de9b0c47c03ae9e"
Oct 30 20:11:34.224846 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.224808    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-token\" (UniqueName: \"kubernetes.io/projected/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-token\") pod \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") "
Oct 30 20:11:34.224968 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.224900    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-envoy\") pod \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") "
Oct 30 20:11:34.225048 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.225006    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-credential-socket\") pod \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") "
Oct 30 20:11:34.225108 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.225085    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-workload-socket\") pod \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") "
Oct 30 20:11:34.225393 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.225190    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-kube-api-access-pv467\" (UniqueName: \"kubernetes.io/projected/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-kube-api-access-pv467\") pod \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") "
Oct 30 20:11:34.225393 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.225282    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-workload-certs\") pod \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") "
Oct 30 20:11:34.225393 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.225337    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-data\") pod \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") "
Oct 30 20:11:34.225393 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.225376    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istiod-ca-cert\") pod \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") "
Oct 30 20:11:34.225656 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.225407    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-podinfo\") pod \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\" (UID: \"4ddd1055-47a1-4c05-bd03-ff3c493f32a7\") "
Oct 30 20:11:34.225931 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.225762    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-credential-socket" pod "4ddd1055-47a1-4c05-bd03-ff3c493f32a7" (UID: "4ddd1055-47a1-4c05-bd03-ff3c493f32a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:34.226116 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.226064    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-data" pod "4ddd1055-47a1-4c05-bd03-ff3c493f32a7" (UID: "4ddd1055-47a1-4c05-bd03-ff3c493f32a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:34.226202 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.226126    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-workload-certs" pod "4ddd1055-47a1-4c05-bd03-ff3c493f32a7" (UID: "4ddd1055-47a1-4c05-bd03-ff3c493f32a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:34.226202 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.226188    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-workload-socket" pod "4ddd1055-47a1-4c05-bd03-ff3c493f32a7" (UID: "4ddd1055-47a1-4c05-bd03-ff3c493f32a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:34.226671 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.226322    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istiod-ca-cert" pod "4ddd1055-47a1-4c05-bd03-ff3c493f32a7" (UID: "4ddd1055-47a1-4c05-bd03-ff3c493f32a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:34.226671 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.226574    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.226671 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.226599    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.226671 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.226613    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.226671 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.226626    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.226671 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.226641    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.227496 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.227472    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-token" pod "4ddd1055-47a1-4c05-bd03-ff3c493f32a7" (UID: "4ddd1055-47a1-4c05-bd03-ff3c493f32a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:34.228468 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.228429    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-podinfo" pod "4ddd1055-47a1-4c05-bd03-ff3c493f32a7" (UID: "4ddd1055-47a1-4c05-bd03-ff3c493f32a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:11:34.230104 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.230082    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-kube-api-access-pv467" pod "4ddd1055-47a1-4c05-bd03-ff3c493f32a7" (UID: "4ddd1055-47a1-4c05-bd03-ff3c493f32a7"). InnerVolumeSpecName "kube-api-access-pv467". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:34.230355 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.230338    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-envoy" pod "4ddd1055-47a1-4c05-bd03-ff3c493f32a7" (UID: "4ddd1055-47a1-4c05-bd03-ff3c493f32a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:34.231776 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.231758    5408 scope.go:122] "RemoveContainer" containerID="7dc0c20e82b2261a27efba3bb9fabb0e26e4254dbb3737f63de9b0c47c03ae9e"
Oct 30 20:11:34.232138 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:11:34.232117    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc0c20e82b2261a27efba3bb9fabb0e26e4254dbb3737f63de9b0c47c03ae9e\": container with ID starting with 7dc0c20e82b2261a27efba3bb9fabb0e26e4254dbb3737f63de9b0c47c03ae9e not found: ID does not exist" containerID="7dc0c20e82b2261a27efba3bb9fabb0e26e4254dbb3737f63de9b0c47c03ae9e"
Oct 30 20:11:34.232222 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.232149    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc0c20e82b2261a27efba3bb9fabb0e26e4254dbb3737f63de9b0c47c03ae9e"} err="failed to get container status \"7dc0c20e82b2261a27efba3bb9fabb0e26e4254dbb3737f63de9b0c47c03ae9e\": rpc error: code = NotFound desc = could not find container \"7dc0c20e82b2261a27efba3bb9fabb0e26e4254dbb3737f63de9b0c47c03ae9e\": container with ID starting with 7dc0c20e82b2261a27efba3bb9fabb0e26e4254dbb3737f63de9b0c47c03ae9e not found: ID does not exist"
Oct 30 20:11:34.328289 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.328255    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.328289 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.328284    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.328289 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.328296    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv467\" (UniqueName: \"kubernetes.io/projected/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-kube-api-access-pv467\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.328592 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.328305    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddd1055-47a1-4c05-bd03-ff3c493f32a7-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.509067 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.509000    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"]
Oct 30 20:11:34.534837 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.534806    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h"]
Oct 30 20:11:34.597869 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.597836    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"]
Oct 30 20:11:34.617275 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:34.617246    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-6gkd9"]
Oct 30 20:11:35.558316 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:35.558278    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ddd1055-47a1-4c05-bd03-ff3c493f32a7" path="/var/lib/kubelet/pods/4ddd1055-47a1-4c05-bd03-ff3c493f32a7/volumes"
Oct 30 20:11:37.585287 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:37.585240    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2" podUID="d3cbd0cd-db30-4181-9271-fab50895d832" containerName="main" containerID="cri-o://4ffc965ca80919565296fbcdfeaab88b1d99309271dcbac45f553d1e9f7ae4eb" gracePeriod=45
Oct 30 20:11:37.880978 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:37.880953    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:11:37.918662 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:37.918627    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-dshm\") pod \"d3cbd0cd-db30-4181-9271-fab50895d832\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") "
Oct 30 20:11:37.918829 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:37.918769    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3cbd0cd-db30-4181-9271-fab50895d832-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3cbd0cd-db30-4181-9271-fab50895d832-tls-certs\") pod \"d3cbd0cd-db30-4181-9271-fab50895d832\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") "
Oct 30 20:11:37.919215 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:37.918965    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-tmp-dir\") pod \"d3cbd0cd-db30-4181-9271-fab50895d832\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") "
Oct 30 20:11:37.919215 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:37.919101    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3cbd0cd-db30-4181-9271-fab50895d832-kube-api-access-8jsvl\" (UniqueName: \"kubernetes.io/projected/d3cbd0cd-db30-4181-9271-fab50895d832-kube-api-access-8jsvl\") pod \"d3cbd0cd-db30-4181-9271-fab50895d832\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") "
Oct 30 20:11:37.919215 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:37.919161    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-home\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-home\") pod \"d3cbd0cd-db30-4181-9271-fab50895d832\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") "
Oct 30 20:11:37.919215 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:37.919194    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-model-cache\") pod \"d3cbd0cd-db30-4181-9271-fab50895d832\" (UID: \"d3cbd0cd-db30-4181-9271-fab50895d832\") "
Oct 30 20:11:37.919837 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:37.919672    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-home" pod "d3cbd0cd-db30-4181-9271-fab50895d832" (UID: "d3cbd0cd-db30-4181-9271-fab50895d832"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:37.919837 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:37.919799    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-model-cache" pod "d3cbd0cd-db30-4181-9271-fab50895d832" (UID: "d3cbd0cd-db30-4181-9271-fab50895d832"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:37.920069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:37.920020    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-tmp-dir" pod "d3cbd0cd-db30-4181-9271-fab50895d832" (UID: "d3cbd0cd-db30-4181-9271-fab50895d832"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:37.922667 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:37.921558    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:37.922667 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:37.921592    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:37.922667 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:37.921605    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:37.922667 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:37.922132    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3cbd0cd-db30-4181-9271-fab50895d832-tls-certs" pod "d3cbd0cd-db30-4181-9271-fab50895d832" (UID: "d3cbd0cd-db30-4181-9271-fab50895d832"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:37.922667 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:37.922260    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-dshm" pod "d3cbd0cd-db30-4181-9271-fab50895d832" (UID: "d3cbd0cd-db30-4181-9271-fab50895d832"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:37.923888 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:37.923861    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3cbd0cd-db30-4181-9271-fab50895d832-kube-api-access-8jsvl" pod "d3cbd0cd-db30-4181-9271-fab50895d832" (UID: "d3cbd0cd-db30-4181-9271-fab50895d832"). InnerVolumeSpecName "kube-api-access-8jsvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:38.022717 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:38.022684    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jsvl\" (UniqueName: \"kubernetes.io/projected/d3cbd0cd-db30-4181-9271-fab50895d832-kube-api-access-8jsvl\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:38.022717 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:38.022717    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3cbd0cd-db30-4181-9271-fab50895d832-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:38.022914 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:38.022732    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3cbd0cd-db30-4181-9271-fab50895d832-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:38.325412 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:38.325380    5408 generic.go:356] "Generic (PLEG): container finished" podID="d3cbd0cd-db30-4181-9271-fab50895d832" containerID="4ffc965ca80919565296fbcdfeaab88b1d99309271dcbac45f553d1e9f7ae4eb" exitCode=0
Oct 30 20:11:38.325576 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:38.325424    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"
Oct 30 20:11:38.325576 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:38.325458    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2" event={"ID":"d3cbd0cd-db30-4181-9271-fab50895d832","Type":"ContainerDied","Data":"4ffc965ca80919565296fbcdfeaab88b1d99309271dcbac45f553d1e9f7ae4eb"}
Oct 30 20:11:38.325576 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:38.325500    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2" event={"ID":"d3cbd0cd-db30-4181-9271-fab50895d832","Type":"ContainerDied","Data":"1beb1c2cfba162011946d5272d476298ee578034706131b2a2b801c6a62c9ede"}
Oct 30 20:11:38.325576 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:38.325524    5408 scope.go:122] "RemoveContainer" containerID="4ffc965ca80919565296fbcdfeaab88b1d99309271dcbac45f553d1e9f7ae4eb"
Oct 30 20:11:38.344683 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:38.344659    5408 scope.go:122] "RemoveContainer" containerID="4ffc965ca80919565296fbcdfeaab88b1d99309271dcbac45f553d1e9f7ae4eb"
Oct 30 20:11:38.344983 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:11:38.344953    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ffc965ca80919565296fbcdfeaab88b1d99309271dcbac45f553d1e9f7ae4eb\": container with ID starting with 4ffc965ca80919565296fbcdfeaab88b1d99309271dcbac45f553d1e9f7ae4eb not found: ID does not exist" containerID="4ffc965ca80919565296fbcdfeaab88b1d99309271dcbac45f553d1e9f7ae4eb"
Oct 30 20:11:38.345091 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:38.344992    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ffc965ca80919565296fbcdfeaab88b1d99309271dcbac45f553d1e9f7ae4eb"} err="failed to get container status \"4ffc965ca80919565296fbcdfeaab88b1d99309271dcbac45f553d1e9f7ae4eb\": rpc error: code = NotFound desc = could not find container \"4ffc965ca80919565296fbcdfeaab88b1d99309271dcbac45f553d1e9f7ae4eb\": container with ID starting with 4ffc965ca80919565296fbcdfeaab88b1d99309271dcbac45f553d1e9f7ae4eb not found: ID does not exist"
Oct 30 20:11:38.354059 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:38.354022    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"]
Oct 30 20:11:38.358701 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:38.358678    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2"]
Oct 30 20:11:38.870923 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:38.870881    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-4f244d/test-model-4f244d-kserve-6d9d7db879-jwrd2" podUID="d3cbd0cd-db30-4181-9271-fab50895d832" containerName="main" probeResult="failure" output="Get \"https://10.134.0.154:8000/ready\": context deadline exceeded"
Oct 30 20:11:39.200777 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.200698    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt" podUID="90d486b1-ecce-40a9-8353-04127a5111df" containerName="main" containerID="cri-o://b6eb2f9893863385ef82ae3b2cba570751cd78d69928011c4309985abe200de4" gracePeriod=45
Oct 30 20:11:39.335724 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.335692    5408 generic.go:356] "Generic (PLEG): container finished" podID="90d486b1-ecce-40a9-8353-04127a5111df" containerID="b6eb2f9893863385ef82ae3b2cba570751cd78d69928011c4309985abe200de4" exitCode=0
Oct 30 20:11:39.335885 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.335833    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt" event={"ID":"90d486b1-ecce-40a9-8353-04127a5111df","Type":"ContainerDied","Data":"b6eb2f9893863385ef82ae3b2cba570751cd78d69928011c4309985abe200de4"}
Oct 30 20:11:39.595966 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.595933    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:11:39.606067 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.606016    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3cbd0cd-db30-4181-9271-fab50895d832" path="/var/lib/kubelet/pods/d3cbd0cd-db30-4181-9271-fab50895d832/volumes"
Oct 30 20:11:39.670851 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:11:39.670809    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:11:39.790368 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.790326    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d486b1-ecce-40a9-8353-04127a5111df-kube-api-access-x4lz7\" (UniqueName: \"kubernetes.io/projected/90d486b1-ecce-40a9-8353-04127a5111df-kube-api-access-x4lz7\") pod \"90d486b1-ecce-40a9-8353-04127a5111df\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") "
Oct 30 20:11:39.790577 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.790559    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90d486b1-ecce-40a9-8353-04127a5111df-tls-certs\" (UniqueName: \"kubernetes.io/secret/90d486b1-ecce-40a9-8353-04127a5111df-tls-certs\") pod \"90d486b1-ecce-40a9-8353-04127a5111df\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") "
Oct 30 20:11:39.790694 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.790673    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-dshm\") pod \"90d486b1-ecce-40a9-8353-04127a5111df\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") "
Oct 30 20:11:39.790811 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.790715    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-home\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-home\") pod \"90d486b1-ecce-40a9-8353-04127a5111df\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") "
Oct 30 20:11:39.790811 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.790757    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-model-cache\") pod \"90d486b1-ecce-40a9-8353-04127a5111df\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") "
Oct 30 20:11:39.790923 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.790869    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-tmp-dir\") pod \"90d486b1-ecce-40a9-8353-04127a5111df\" (UID: \"90d486b1-ecce-40a9-8353-04127a5111df\") "
Oct 30 20:11:39.790997 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.790971    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-home" pod "90d486b1-ecce-40a9-8353-04127a5111df" (UID: "90d486b1-ecce-40a9-8353-04127a5111df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:39.791231 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.791162    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-model-cache" pod "90d486b1-ecce-40a9-8353-04127a5111df" (UID: "90d486b1-ecce-40a9-8353-04127a5111df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:39.791342 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.791244    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-tmp-dir" pod "90d486b1-ecce-40a9-8353-04127a5111df" (UID: "90d486b1-ecce-40a9-8353-04127a5111df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:39.791615 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.791593    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:39.791677 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.791623    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:39.791677 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.791639    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:39.792975 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.792948    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d486b1-ecce-40a9-8353-04127a5111df-kube-api-access-x4lz7" pod "90d486b1-ecce-40a9-8353-04127a5111df" (UID: "90d486b1-ecce-40a9-8353-04127a5111df"). InnerVolumeSpecName "kube-api-access-x4lz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:39.793253 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.793232    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d486b1-ecce-40a9-8353-04127a5111df-tls-certs" pod "90d486b1-ecce-40a9-8353-04127a5111df" (UID: "90d486b1-ecce-40a9-8353-04127a5111df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:39.793344 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.793238    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-dshm" pod "90d486b1-ecce-40a9-8353-04127a5111df" (UID: "90d486b1-ecce-40a9-8353-04127a5111df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:39.893881 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.893838    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d486b1-ecce-40a9-8353-04127a5111df-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:39.893881 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.893879    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d486b1-ecce-40a9-8353-04127a5111df-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:39.894472 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:39.893897    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4lz7\" (UniqueName: \"kubernetes.io/projected/90d486b1-ecce-40a9-8353-04127a5111df-kube-api-access-x4lz7\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:40.311084 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:40.296124    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h" podUID="195cd4cc-15b4-470c-b139-f0dd93530204" containerName="payload-pre-processing" containerID="cri-o://31c1c6a4e53e1c84b20156561113800671c01c139f532a768b52779ab5f26a84" gracePeriod=25
Oct 30 20:11:40.428835 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:40.428785    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt" event={"ID":"90d486b1-ecce-40a9-8353-04127a5111df","Type":"ContainerDied","Data":"56ac3c2f32403cfc08769e498bbb26a9b8c35c897a9d56b4e9779631c59e67a3"}
Oct 30 20:11:40.429116 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:40.429097    5408 scope.go:122] "RemoveContainer" containerID="b6eb2f9893863385ef82ae3b2cba570751cd78d69928011c4309985abe200de4"
Oct 30 20:11:40.429602 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:40.429569    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"
Oct 30 20:11:40.498693 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:40.498661    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"]
Oct 30 20:11:40.498884 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:40.498716    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d7a733/test-model-d7a733-kserve-5d55d59dcc-rbfbt"]
Oct 30 20:11:40.702968 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:40.702931    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h"
Oct 30 20:11:40.820596 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:40.820560    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/195cd4cc-15b4-470c-b139-f0dd93530204-kube-api-access-l2k7c\" (UniqueName: \"kubernetes.io/projected/195cd4cc-15b4-470c-b139-f0dd93530204-kube-api-access-l2k7c\") pod \"195cd4cc-15b4-470c-b139-f0dd93530204\" (UID: \"195cd4cc-15b4-470c-b139-f0dd93530204\") "
Oct 30 20:11:40.820756 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:40.820712    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/195cd4cc-15b4-470c-b139-f0dd93530204-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195cd4cc-15b4-470c-b139-f0dd93530204-plugins-config-volume\") pod \"195cd4cc-15b4-470c-b139-f0dd93530204\" (UID: \"195cd4cc-15b4-470c-b139-f0dd93530204\") "
Oct 30 20:11:40.821115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:40.821089    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195cd4cc-15b4-470c-b139-f0dd93530204-plugins-config-volume" pod "195cd4cc-15b4-470c-b139-f0dd93530204" (UID: "195cd4cc-15b4-470c-b139-f0dd93530204"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:40.821448 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:40.821429    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195cd4cc-15b4-470c-b139-f0dd93530204-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:40.822836 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:40.822815    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195cd4cc-15b4-470c-b139-f0dd93530204-kube-api-access-l2k7c" pod "195cd4cc-15b4-470c-b139-f0dd93530204" (UID: "195cd4cc-15b4-470c-b139-f0dd93530204"). InnerVolumeSpecName "kube-api-access-l2k7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:40.926056 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:40.924269    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2k7c\" (UniqueName: \"kubernetes.io/projected/195cd4cc-15b4-470c-b139-f0dd93530204-kube-api-access-l2k7c\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:41.016562 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:41.016529    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl"]
Oct 30 20:11:41.028512 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:41.028468    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477"]
Oct 30 20:11:41.478056 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:41.476051    5408 generic.go:356] "Generic (PLEG): container finished" podID="195cd4cc-15b4-470c-b139-f0dd93530204" containerID="31c1c6a4e53e1c84b20156561113800671c01c139f532a768b52779ab5f26a84" exitCode=0
Oct 30 20:11:41.478056 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:41.476170    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h" event={"ID":"195cd4cc-15b4-470c-b139-f0dd93530204","Type":"ContainerDied","Data":"31c1c6a4e53e1c84b20156561113800671c01c139f532a768b52779ab5f26a84"}
Oct 30 20:11:41.478056 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:41.476219    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h" event={"ID":"195cd4cc-15b4-470c-b139-f0dd93530204","Type":"ContainerDied","Data":"9450686144a19381ffd71afce2a3aada8dde6a650353319d60d2e8016099662e"}
Oct 30 20:11:41.478056 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:41.476251    5408 scope.go:122] "RemoveContainer" containerID="31c1c6a4e53e1c84b20156561113800671c01c139f532a768b52779ab5f26a84"
Oct 30 20:11:41.478056 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:41.476359    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h"
Oct 30 20:11:41.585248 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:41.583740    5408 scope.go:122] "RemoveContainer" containerID="31c1c6a4e53e1c84b20156561113800671c01c139f532a768b52779ab5f26a84"
Oct 30 20:11:41.595238 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:11:41.592188    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31c1c6a4e53e1c84b20156561113800671c01c139f532a768b52779ab5f26a84\": container with ID starting with 31c1c6a4e53e1c84b20156561113800671c01c139f532a768b52779ab5f26a84 not found: ID does not exist" containerID="31c1c6a4e53e1c84b20156561113800671c01c139f532a768b52779ab5f26a84"
Oct 30 20:11:41.595238 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:41.592234    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c1c6a4e53e1c84b20156561113800671c01c139f532a768b52779ab5f26a84"} err="failed to get container status \"31c1c6a4e53e1c84b20156561113800671c01c139f532a768b52779ab5f26a84\": rpc error: code = NotFound desc = could not find container \"31c1c6a4e53e1c84b20156561113800671c01c139f532a768b52779ab5f26a84\": container with ID starting with 31c1c6a4e53e1c84b20156561113800671c01c139f532a768b52779ab5f26a84 not found: ID does not exist"
Oct 30 20:11:41.603050 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:41.600384    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90d486b1-ecce-40a9-8353-04127a5111df" path="/var/lib/kubelet/pods/90d486b1-ecce-40a9-8353-04127a5111df/volumes"
Oct 30 20:11:41.603050 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:41.600959    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h"]
Oct 30 20:11:41.607056 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:41.606641    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a7bb6c-84bbd946c-8wq2h"]
Oct 30 20:11:42.330055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:42.328306    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"]
Oct 30 20:11:42.330055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:42.328560    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg" podUID="f99c11d2-540e-434a-8271-92ea41a6b437" containerName="istio-proxy" containerID="cri-o://b911e63d830ac74f035c3c11fe0ef04eb5995e3bc7a44fa85247fb049e4a565e" gracePeriod=30
Oct 30 20:11:43.536085 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:43.536012    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="195cd4cc-15b4-470c-b139-f0dd93530204" path="/var/lib/kubelet/pods/195cd4cc-15b4-470c-b139-f0dd93530204/volumes"
Oct 30 20:11:45.576145 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:45.576111    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd" podUID="b19efa61-42ee-40ce-98b4-6b57dd46b3c1" containerName="main" containerID="cri-o://4f0bb9358d5261433674e74212145cec12696d210839231ac54e6def43b495dc" gracePeriod=45
Oct 30 20:11:45.886481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:45.886454    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:11:45.955203 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:45.954856    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-kube-api-access-jz998\" (UniqueName: \"kubernetes.io/projected/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-kube-api-access-jz998\") pod \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") "
Oct 30 20:11:45.955203 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:45.954931    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-dshm\") pod \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") "
Oct 30 20:11:45.955203 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:45.954965    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-model-cache\") pod \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") "
Oct 30 20:11:45.965053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:45.963189    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-model-cache" pod "b19efa61-42ee-40ce-98b4-6b57dd46b3c1" (UID: "b19efa61-42ee-40ce-98b4-6b57dd46b3c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:45.965361 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:45.965335    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-tls-certs\") pod \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") "
Oct 30 20:11:45.974473 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:45.974416    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-home" pod "b19efa61-42ee-40ce-98b4-6b57dd46b3c1" (UID: "b19efa61-42ee-40ce-98b4-6b57dd46b3c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:45.976334 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:45.976302    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-home\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-home\") pod \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") "
Oct 30 20:11:45.976577 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:45.976563    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-tmp-dir\") pod \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\" (UID: \"b19efa61-42ee-40ce-98b4-6b57dd46b3c1\") "
Oct 30 20:11:45.976971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:45.976853    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-tmp-dir" pod "b19efa61-42ee-40ce-98b4-6b57dd46b3c1" (UID: "b19efa61-42ee-40ce-98b4-6b57dd46b3c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:45.977832 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:45.977806    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:45.977961 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:45.977948    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:45.978097 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:45.978083    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:45.986379 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:45.985903    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-kube-api-access-jz998" pod "b19efa61-42ee-40ce-98b4-6b57dd46b3c1" (UID: "b19efa61-42ee-40ce-98b4-6b57dd46b3c1"). InnerVolumeSpecName "kube-api-access-jz998". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:45.986379 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:45.986330    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-dshm" pod "b19efa61-42ee-40ce-98b4-6b57dd46b3c1" (UID: "b19efa61-42ee-40ce-98b4-6b57dd46b3c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:45.994776 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:45.994744    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-tls-certs" pod "b19efa61-42ee-40ce-98b4-6b57dd46b3c1" (UID: "b19efa61-42ee-40ce-98b4-6b57dd46b3c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:46.017069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.016999    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl" podUID="c27eb3c2-6159-459c-8ef1-6ef50c9b62cc" containerName="payload-processing" containerID="cri-o://47a3793ee7e943f760f895c3ec1c3176dd4e9efa8034a77c7bfffb002b225c69" gracePeriod=25
Oct 30 20:11:46.029664 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.029566    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477" podUID="38fa73eb-88a7-4868-8e33-a407edc23aef" containerName="payload-pre-processing" containerID="cri-o://c57a296e9ed43b4e4c1ba2e4c1978a5f289545efcaea809a57310cd17b7ca7eb" gracePeriod=25
Oct 30 20:11:46.080180 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.080152    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz998\" (UniqueName: \"kubernetes.io/projected/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-kube-api-access-jz998\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:46.080180 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.080178    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:46.080180 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.080188    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19efa61-42ee-40ce-98b4-6b57dd46b3c1-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:46.594392 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.591527    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl"
Oct 30 20:11:46.598086 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.597147    5408 generic.go:356] "Generic (PLEG): container finished" podID="38fa73eb-88a7-4868-8e33-a407edc23aef" containerID="c57a296e9ed43b4e4c1ba2e4c1978a5f289545efcaea809a57310cd17b7ca7eb" exitCode=0
Oct 30 20:11:46.598086 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.597200    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477" event={"ID":"38fa73eb-88a7-4868-8e33-a407edc23aef","Type":"ContainerDied","Data":"c57a296e9ed43b4e4c1ba2e4c1978a5f289545efcaea809a57310cd17b7ca7eb"}
Oct 30 20:11:46.599573 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.599546    5408 generic.go:356] "Generic (PLEG): container finished" podID="c27eb3c2-6159-459c-8ef1-6ef50c9b62cc" containerID="47a3793ee7e943f760f895c3ec1c3176dd4e9efa8034a77c7bfffb002b225c69" exitCode=0
Oct 30 20:11:46.599698 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.599636    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl" event={"ID":"c27eb3c2-6159-459c-8ef1-6ef50c9b62cc","Type":"ContainerDied","Data":"47a3793ee7e943f760f895c3ec1c3176dd4e9efa8034a77c7bfffb002b225c69"}
Oct 30 20:11:46.599698 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.599671    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl" event={"ID":"c27eb3c2-6159-459c-8ef1-6ef50c9b62cc","Type":"ContainerDied","Data":"31481a7526bf5903a8d121b49779b6d2089783f03e9add1b5ca51bc44888659c"}
Oct 30 20:11:46.599698 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.599693    5408 scope.go:122] "RemoveContainer" containerID="47a3793ee7e943f760f895c3ec1c3176dd4e9efa8034a77c7bfffb002b225c69"
Oct 30 20:11:46.599884 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.599791    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl"
Oct 30 20:11:46.603405 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.603326    5408 generic.go:356] "Generic (PLEG): container finished" podID="b19efa61-42ee-40ce-98b4-6b57dd46b3c1" containerID="4f0bb9358d5261433674e74212145cec12696d210839231ac54e6def43b495dc" exitCode=0
Oct 30 20:11:46.603508 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.603409    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd" event={"ID":"b19efa61-42ee-40ce-98b4-6b57dd46b3c1","Type":"ContainerDied","Data":"4f0bb9358d5261433674e74212145cec12696d210839231ac54e6def43b495dc"}
Oct 30 20:11:46.603508 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.603432    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd" event={"ID":"b19efa61-42ee-40ce-98b4-6b57dd46b3c1","Type":"ContainerDied","Data":"bb1085608fdde35f3fff6f16484adbca54a64aa653c5be0c234dc459282274d3"}
Oct 30 20:11:46.603508 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.603492    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"
Oct 30 20:11:46.639652 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.639630    5408 scope.go:122] "RemoveContainer" containerID="47a3793ee7e943f760f895c3ec1c3176dd4e9efa8034a77c7bfffb002b225c69"
Oct 30 20:11:46.639966 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:11:46.639939    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a3793ee7e943f760f895c3ec1c3176dd4e9efa8034a77c7bfffb002b225c69\": container with ID starting with 47a3793ee7e943f760f895c3ec1c3176dd4e9efa8034a77c7bfffb002b225c69 not found: ID does not exist" containerID="47a3793ee7e943f760f895c3ec1c3176dd4e9efa8034a77c7bfffb002b225c69"
Oct 30 20:11:46.640074 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.639972    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a3793ee7e943f760f895c3ec1c3176dd4e9efa8034a77c7bfffb002b225c69"} err="failed to get container status \"47a3793ee7e943f760f895c3ec1c3176dd4e9efa8034a77c7bfffb002b225c69\": rpc error: code = NotFound desc = could not find container \"47a3793ee7e943f760f895c3ec1c3176dd4e9efa8034a77c7bfffb002b225c69\": container with ID starting with 47a3793ee7e943f760f895c3ec1c3176dd4e9efa8034a77c7bfffb002b225c69 not found: ID does not exist"
Oct 30 20:11:46.640074 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.639989    5408 scope.go:122] "RemoveContainer" containerID="4f0bb9358d5261433674e74212145cec12696d210839231ac54e6def43b495dc"
Oct 30 20:11:46.643752 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.643729    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477"
Oct 30 20:11:46.651620 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.651599    5408 scope.go:122] "RemoveContainer" containerID="4f0bb9358d5261433674e74212145cec12696d210839231ac54e6def43b495dc"
Oct 30 20:11:46.651902 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:11:46.651883    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0bb9358d5261433674e74212145cec12696d210839231ac54e6def43b495dc\": container with ID starting with 4f0bb9358d5261433674e74212145cec12696d210839231ac54e6def43b495dc not found: ID does not exist" containerID="4f0bb9358d5261433674e74212145cec12696d210839231ac54e6def43b495dc"
Oct 30 20:11:46.651992 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.651910    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0bb9358d5261433674e74212145cec12696d210839231ac54e6def43b495dc"} err="failed to get container status \"4f0bb9358d5261433674e74212145cec12696d210839231ac54e6def43b495dc\": rpc error: code = NotFound desc = could not find container \"4f0bb9358d5261433674e74212145cec12696d210839231ac54e6def43b495dc\": container with ID starting with 4f0bb9358d5261433674e74212145cec12696d210839231ac54e6def43b495dc not found: ID does not exist"
Oct 30 20:11:46.695907 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.695874    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c27eb3c2-6159-459c-8ef1-6ef50c9b62cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27eb3c2-6159-459c-8ef1-6ef50c9b62cc-plugins-config-volume\") pod \"c27eb3c2-6159-459c-8ef1-6ef50c9b62cc\" (UID: \"c27eb3c2-6159-459c-8ef1-6ef50c9b62cc\") "
Oct 30 20:11:46.696079 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.695969    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38fa73eb-88a7-4868-8e33-a407edc23aef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38fa73eb-88a7-4868-8e33-a407edc23aef-plugins-config-volume\") pod \"38fa73eb-88a7-4868-8e33-a407edc23aef\" (UID: \"38fa73eb-88a7-4868-8e33-a407edc23aef\") "
Oct 30 20:11:46.696162 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.696105    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27eb3c2-6159-459c-8ef1-6ef50c9b62cc-kube-api-access-6lgzv\" (UniqueName: \"kubernetes.io/projected/c27eb3c2-6159-459c-8ef1-6ef50c9b62cc-kube-api-access-6lgzv\") pod \"c27eb3c2-6159-459c-8ef1-6ef50c9b62cc\" (UID: \"c27eb3c2-6159-459c-8ef1-6ef50c9b62cc\") "
Oct 30 20:11:46.696269 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.696249    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38fa73eb-88a7-4868-8e33-a407edc23aef-kube-api-access-v7mb5\" (UniqueName: \"kubernetes.io/projected/38fa73eb-88a7-4868-8e33-a407edc23aef-kube-api-access-v7mb5\") pod \"38fa73eb-88a7-4868-8e33-a407edc23aef\" (UID: \"38fa73eb-88a7-4868-8e33-a407edc23aef\") "
Oct 30 20:11:46.696401 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.696369    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27eb3c2-6159-459c-8ef1-6ef50c9b62cc-plugins-config-volume" pod "c27eb3c2-6159-459c-8ef1-6ef50c9b62cc" (UID: "c27eb3c2-6159-459c-8ef1-6ef50c9b62cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:46.696517 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.696426    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38fa73eb-88a7-4868-8e33-a407edc23aef-plugins-config-volume" pod "38fa73eb-88a7-4868-8e33-a407edc23aef" (UID: "38fa73eb-88a7-4868-8e33-a407edc23aef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:46.696985 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.696956    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27eb3c2-6159-459c-8ef1-6ef50c9b62cc-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:46.696985 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.696983    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38fa73eb-88a7-4868-8e33-a407edc23aef-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:46.698597 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.698577    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fa73eb-88a7-4868-8e33-a407edc23aef-kube-api-access-v7mb5" pod "38fa73eb-88a7-4868-8e33-a407edc23aef" (UID: "38fa73eb-88a7-4868-8e33-a407edc23aef"). InnerVolumeSpecName "kube-api-access-v7mb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:46.698815 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.698798    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27eb3c2-6159-459c-8ef1-6ef50c9b62cc-kube-api-access-6lgzv" pod "c27eb3c2-6159-459c-8ef1-6ef50c9b62cc" (UID: "c27eb3c2-6159-459c-8ef1-6ef50c9b62cc"). InnerVolumeSpecName "kube-api-access-6lgzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:46.711776 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.711758    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"]
Oct 30 20:11:46.747374 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.747350    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-77b4df/test-model-77b4df-kserve-95796d7f-2t7dd"]
Oct 30 20:11:46.798390 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.798359    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lgzv\" (UniqueName: \"kubernetes.io/projected/c27eb3c2-6159-459c-8ef1-6ef50c9b62cc-kube-api-access-6lgzv\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:46.798390 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.798389    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7mb5\" (UniqueName: \"kubernetes.io/projected/38fa73eb-88a7-4868-8e33-a407edc23aef-kube-api-access-v7mb5\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:46.932047 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.932008    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl"]
Oct 30 20:11:46.935212 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:46.935185    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4f244d-6794ddbd48-vkcnl"]
Oct 30 20:11:47.081520 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.081486    5408 scope.go:122] "RemoveContainer" containerID="c57a296e9ed43b4e4c1ba2e4c1978a5f289545efcaea809a57310cd17b7ca7eb"
Oct 30 20:11:47.557480 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.557446    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b19efa61-42ee-40ce-98b4-6b57dd46b3c1" path="/var/lib/kubelet/pods/b19efa61-42ee-40ce-98b4-6b57dd46b3c1/volumes"
Oct 30 20:11:47.558198 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.558173    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c27eb3c2-6159-459c-8ef1-6ef50c9b62cc" path="/var/lib/kubelet/pods/c27eb3c2-6159-459c-8ef1-6ef50c9b62cc/volumes"
Oct 30 20:11:47.742651 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.736412    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477" event={"ID":"38fa73eb-88a7-4868-8e33-a407edc23aef","Type":"ContainerDied","Data":"4e569b94d6374c61050087b844893aeed3d0dea623d1239c04d8545d8a9e01e8"}
Oct 30 20:11:47.742651 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.741445    5408 generic.go:356] "Generic (PLEG): container finished" podID="f99c11d2-540e-434a-8271-92ea41a6b437" containerID="b911e63d830ac74f035c3c11fe0ef04eb5995e3bc7a44fa85247fb049e4a565e" exitCode=0
Oct 30 20:11:47.742651 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.741562    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg" event={"ID":"f99c11d2-540e-434a-8271-92ea41a6b437","Type":"ContainerDied","Data":"b911e63d830ac74f035c3c11fe0ef04eb5995e3bc7a44fa85247fb049e4a565e"}
Oct 30 20:11:47.742651 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.741615    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg" event={"ID":"f99c11d2-540e-434a-8271-92ea41a6b437","Type":"ContainerDied","Data":"836ea83c0c3401ab9a4f6d6ae24cf8276927c97f1b34121195932bb5d7a4623a"}
Oct 30 20:11:47.742651 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.741640    5408 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="836ea83c0c3401ab9a4f6d6ae24cf8276927c97f1b34121195932bb5d7a4623a"
Oct 30 20:11:47.742651 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.741669    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477"
Oct 30 20:11:47.742651 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.741689    5408 scope.go:122] "RemoveContainer" containerID="f0ec32cfcaf47fd6034c05d17dbafedead82651b9fa951537c0397927bab4d32"
Oct 30 20:11:47.762413 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.762383    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:11:47.785765 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.785736    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477"]
Oct 30 20:11:47.794197 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.794177    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4f244d-9b6c59f8c-dk477"]
Oct 30 20:11:47.818153 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.818123    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-credential-socket\") pod \"f99c11d2-540e-434a-8271-92ea41a6b437\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") "
Oct 30 20:11:47.818301 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.818194    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-istio-data\") pod \"f99c11d2-540e-434a-8271-92ea41a6b437\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") "
Oct 30 20:11:47.818301 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.818255    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f99c11d2-540e-434a-8271-92ea41a6b437-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99c11d2-540e-434a-8271-92ea41a6b437-istiod-ca-cert\") pod \"f99c11d2-540e-434a-8271-92ea41a6b437\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") "
Oct 30 20:11:47.818459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.818326    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f99c11d2-540e-434a-8271-92ea41a6b437-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99c11d2-540e-434a-8271-92ea41a6b437-istio-podinfo\") pod \"f99c11d2-540e-434a-8271-92ea41a6b437\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") "
Oct 30 20:11:47.818459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.818358    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-workload-socket\") pod \"f99c11d2-540e-434a-8271-92ea41a6b437\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") "
Oct 30 20:11:47.818459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.818418    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99c11d2-540e-434a-8271-92ea41a6b437-istio-token\" (UniqueName: \"kubernetes.io/projected/f99c11d2-540e-434a-8271-92ea41a6b437-istio-token\") pod \"f99c11d2-540e-434a-8271-92ea41a6b437\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") "
Oct 30 20:11:47.818459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.818446    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99c11d2-540e-434a-8271-92ea41a6b437-kube-api-access-xqjwl\" (UniqueName: \"kubernetes.io/projected/f99c11d2-540e-434a-8271-92ea41a6b437-kube-api-access-xqjwl\") pod \"f99c11d2-540e-434a-8271-92ea41a6b437\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") "
Oct 30 20:11:47.818714 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.818502    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-istio-envoy\") pod \"f99c11d2-540e-434a-8271-92ea41a6b437\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") "
Oct 30 20:11:47.818714 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.818593    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-workload-certs\") pod \"f99c11d2-540e-434a-8271-92ea41a6b437\" (UID: \"f99c11d2-540e-434a-8271-92ea41a6b437\") "
Oct 30 20:11:47.818714 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.818634    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-istio-data" pod "f99c11d2-540e-434a-8271-92ea41a6b437" (UID: "f99c11d2-540e-434a-8271-92ea41a6b437"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:47.819214 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.819175    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-credential-socket" pod "f99c11d2-540e-434a-8271-92ea41a6b437" (UID: "f99c11d2-540e-434a-8271-92ea41a6b437"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:47.819430 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.819408    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:47.819430 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.819418    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-workload-certs" pod "f99c11d2-540e-434a-8271-92ea41a6b437" (UID: "f99c11d2-540e-434a-8271-92ea41a6b437"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:47.819581 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.819444    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:47.819650 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.819601    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-workload-socket" pod "f99c11d2-540e-434a-8271-92ea41a6b437" (UID: "f99c11d2-540e-434a-8271-92ea41a6b437"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:47.821899 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.821840    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-istio-envoy" pod "f99c11d2-540e-434a-8271-92ea41a6b437" (UID: "f99c11d2-540e-434a-8271-92ea41a6b437"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:47.822008 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.821865    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99c11d2-540e-434a-8271-92ea41a6b437-istio-token" pod "f99c11d2-540e-434a-8271-92ea41a6b437" (UID: "f99c11d2-540e-434a-8271-92ea41a6b437"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:47.822135 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.822111    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f99c11d2-540e-434a-8271-92ea41a6b437-istiod-ca-cert" pod "f99c11d2-540e-434a-8271-92ea41a6b437" (UID: "f99c11d2-540e-434a-8271-92ea41a6b437"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:47.823656 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.823625    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99c11d2-540e-434a-8271-92ea41a6b437-kube-api-access-xqjwl" pod "f99c11d2-540e-434a-8271-92ea41a6b437" (UID: "f99c11d2-540e-434a-8271-92ea41a6b437"). InnerVolumeSpecName "kube-api-access-xqjwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:47.824195 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.824171    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f99c11d2-540e-434a-8271-92ea41a6b437-istio-podinfo" pod "f99c11d2-540e-434a-8271-92ea41a6b437" (UID: "f99c11d2-540e-434a-8271-92ea41a6b437"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:11:47.920713 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.920684    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f99c11d2-540e-434a-8271-92ea41a6b437-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:47.920713 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.920712    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f99c11d2-540e-434a-8271-92ea41a6b437-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:47.920713 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.920723    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:47.920713 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.920732    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f99c11d2-540e-434a-8271-92ea41a6b437-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:47.921096 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.920741    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqjwl\" (UniqueName: \"kubernetes.io/projected/f99c11d2-540e-434a-8271-92ea41a6b437-kube-api-access-xqjwl\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:47.921096 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.920751    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:47.921096 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.920759    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f99c11d2-540e-434a-8271-92ea41a6b437-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:47.944750 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.944718    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"]
Oct 30 20:11:47.951195 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:47.951144    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"]
Oct 30 20:11:48.754777 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:48.754744    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"
Oct 30 20:11:48.845053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:48.836077    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"]
Oct 30 20:11:48.845053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:48.840242    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-8fnxg"]
Oct 30 20:11:49.533153 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:49.533113    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main" containerID="cri-o://be85bf2e96ba8a10005fce6b6d4288a5aecb57b96861ce1a0054bfa83cfb207c" gracePeriod=45
Oct 30 20:11:49.535837 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:49.535813    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38fa73eb-88a7-4868-8e33-a407edc23aef" path="/var/lib/kubelet/pods/38fa73eb-88a7-4868-8e33-a407edc23aef/volumes"
Oct 30 20:11:49.536463 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:49.536440    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f99c11d2-540e-434a-8271-92ea41a6b437" path="/var/lib/kubelet/pods/f99c11d2-540e-434a-8271-92ea41a6b437/volumes"
Oct 30 20:11:49.769223 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:49.769190    5408 generic.go:356] "Generic (PLEG): container finished" podID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerID="be85bf2e96ba8a10005fce6b6d4288a5aecb57b96861ce1a0054bfa83cfb207c" exitCode=0
Oct 30 20:11:49.769844 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:49.769228    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" event={"ID":"3a38c258-0d4e-4b70-afec-45f6025fdcc6","Type":"ContainerDied","Data":"be85bf2e96ba8a10005fce6b6d4288a5aecb57b96861ce1a0054bfa83cfb207c"}
Oct 30 20:11:49.953066 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:49.944838    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 20:11:50.069664 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.069611    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-home\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-home\") pod \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") "
Oct 30 20:11:50.069993 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.069962    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a38c258-0d4e-4b70-afec-45f6025fdcc6-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a38c258-0d4e-4b70-afec-45f6025fdcc6-tls-certs\") pod \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") "
Oct 30 20:11:50.070278 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.070249    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a38c258-0d4e-4b70-afec-45f6025fdcc6-kube-api-access-7kqqb\" (UniqueName: \"kubernetes.io/projected/3a38c258-0d4e-4b70-afec-45f6025fdcc6-kube-api-access-7kqqb\") pod \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") "
Oct 30 20:11:50.070487 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.070471    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-tmp-dir\") pod \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") "
Oct 30 20:11:50.070753 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.070735    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-model-cache\") pod \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") "
Oct 30 20:11:50.070915 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.070892    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-dshm\") pod \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\" (UID: \"3a38c258-0d4e-4b70-afec-45f6025fdcc6\") "
Oct 30 20:11:50.095102 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.090249    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-home" pod "3a38c258-0d4e-4b70-afec-45f6025fdcc6" (UID: "3a38c258-0d4e-4b70-afec-45f6025fdcc6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:50.095102 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.090796    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-tmp-dir" pod "3a38c258-0d4e-4b70-afec-45f6025fdcc6" (UID: "3a38c258-0d4e-4b70-afec-45f6025fdcc6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:50.095102 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.092502    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-model-cache" pod "3a38c258-0d4e-4b70-afec-45f6025fdcc6" (UID: "3a38c258-0d4e-4b70-afec-45f6025fdcc6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:50.110844 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.110806    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-dshm" pod "3a38c258-0d4e-4b70-afec-45f6025fdcc6" (UID: "3a38c258-0d4e-4b70-afec-45f6025fdcc6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:50.111094 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.110816    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a38c258-0d4e-4b70-afec-45f6025fdcc6-kube-api-access-7kqqb" pod "3a38c258-0d4e-4b70-afec-45f6025fdcc6" (UID: "3a38c258-0d4e-4b70-afec-45f6025fdcc6"). InnerVolumeSpecName "kube-api-access-7kqqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:50.117546 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.117287    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a38c258-0d4e-4b70-afec-45f6025fdcc6-tls-certs" pod "3a38c258-0d4e-4b70-afec-45f6025fdcc6" (UID: "3a38c258-0d4e-4b70-afec-45f6025fdcc6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:50.194185 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.194112    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:50.194185 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.194150    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:50.194185 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.194165    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a38c258-0d4e-4b70-afec-45f6025fdcc6-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:50.194185 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.194181    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kqqb\" (UniqueName: \"kubernetes.io/projected/3a38c258-0d4e-4b70-afec-45f6025fdcc6-kube-api-access-7kqqb\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:50.194185 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.194196    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:50.194626 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.194211    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a38c258-0d4e-4b70-afec-45f6025fdcc6-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:50.777291 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.777259    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn" event={"ID":"3a38c258-0d4e-4b70-afec-45f6025fdcc6","Type":"ContainerDied","Data":"d531451dfe33f687b6bf57510a8a405ebed678e2cd97f123e942f84817051c57"}
Oct 30 20:11:50.777735 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.777295    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"
Oct 30 20:11:50.777735 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.777319    5408 scope.go:122] "RemoveContainer" containerID="be85bf2e96ba8a10005fce6b6d4288a5aecb57b96861ce1a0054bfa83cfb207c"
Oct 30 20:11:50.851063 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.850975    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"]
Oct 30 20:11:50.861829 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:50.861795    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/distinct-2-w2-d5e3e5-kserve-78fb5bd457-h29tn"]
Oct 30 20:11:51.586099 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:51.575145    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" path="/var/lib/kubelet/pods/3a38c258-0d4e-4b70-afec-45f6025fdcc6/volumes"
Oct 30 20:11:54.803561 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:54.803527    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p"]
Oct 30 20:11:54.833218 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:54.833190    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4"]
Oct 30 20:11:57.100250 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:57.100214    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7"]
Oct 30 20:11:57.116658 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:57.116626    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l"]
Oct 30 20:11:58.323750 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:58.323716    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"]
Oct 30 20:11:58.324525 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:58.324483    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8" podUID="c024192c-6410-4363-89b9-4b0e4fd5e4dc" containerName="maas-api" containerID="cri-o://674a72e698d47113300429da81abd44d46d9689797ce47f42c3c31d30d60ed0e" gracePeriod=30
Oct 30 20:11:58.896079 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:58.896048    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"
Oct 30 20:11:58.899067 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:58.899011    5408 generic.go:356] "Generic (PLEG): container finished" podID="c024192c-6410-4363-89b9-4b0e4fd5e4dc" containerID="674a72e698d47113300429da81abd44d46d9689797ce47f42c3c31d30d60ed0e" exitCode=0
Oct 30 20:11:58.899215 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:58.899072    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8" event={"ID":"c024192c-6410-4363-89b9-4b0e4fd5e4dc","Type":"ContainerDied","Data":"674a72e698d47113300429da81abd44d46d9689797ce47f42c3c31d30d60ed0e"}
Oct 30 20:11:58.899215 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:58.899098    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"
Oct 30 20:11:58.899215 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:58.899140    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8" event={"ID":"c024192c-6410-4363-89b9-4b0e4fd5e4dc","Type":"ContainerDied","Data":"94d07dde771e5c1133f0a5adfad03c0833ff97e59683619ea177b12ac48e2c7e"}
Oct 30 20:11:58.899215 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:58.899174    5408 scope.go:122] "RemoveContainer" containerID="674a72e698d47113300429da81abd44d46d9689797ce47f42c3c31d30d60ed0e"
Oct 30 20:11:58.911273 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:58.911251    5408 scope.go:122] "RemoveContainer" containerID="674a72e698d47113300429da81abd44d46d9689797ce47f42c3c31d30d60ed0e"
Oct 30 20:11:58.911615 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:11:58.911582    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674a72e698d47113300429da81abd44d46d9689797ce47f42c3c31d30d60ed0e\": container with ID starting with 674a72e698d47113300429da81abd44d46d9689797ce47f42c3c31d30d60ed0e not found: ID does not exist" containerID="674a72e698d47113300429da81abd44d46d9689797ce47f42c3c31d30d60ed0e"
Oct 30 20:11:58.911754 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:58.911624    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674a72e698d47113300429da81abd44d46d9689797ce47f42c3c31d30d60ed0e"} err="failed to get container status \"674a72e698d47113300429da81abd44d46d9689797ce47f42c3c31d30d60ed0e\": rpc error: code = NotFound desc = could not find container \"674a72e698d47113300429da81abd44d46d9689797ce47f42c3c31d30d60ed0e\": container with ID starting with 674a72e698d47113300429da81abd44d46d9689797ce47f42c3c31d30d60ed0e not found: ID does not exist"
Oct 30 20:11:59.000995 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:59.000963    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c024192c-6410-4363-89b9-4b0e4fd5e4dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c024192c-6410-4363-89b9-4b0e4fd5e4dc-openshift-service-ca\") pod \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\" (UID: \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\") "
Oct 30 20:11:59.001193 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:59.001073    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c024192c-6410-4363-89b9-4b0e4fd5e4dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c024192c-6410-4363-89b9-4b0e4fd5e4dc-metrics-certs\") pod \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\" (UID: \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\") "
Oct 30 20:11:59.001193 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:59.001174    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c024192c-6410-4363-89b9-4b0e4fd5e4dc-kube-api-access-rn42q\" (UniqueName: \"kubernetes.io/projected/c024192c-6410-4363-89b9-4b0e4fd5e4dc-kube-api-access-rn42q\") pod \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\" (UID: \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\") "
Oct 30 20:11:59.001348 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:59.001233    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c024192c-6410-4363-89b9-4b0e4fd5e4dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c024192c-6410-4363-89b9-4b0e4fd5e4dc-maas-api-tls\") pod \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\" (UID: \"c024192c-6410-4363-89b9-4b0e4fd5e4dc\") "
Oct 30 20:11:59.001455 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:59.001426    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c024192c-6410-4363-89b9-4b0e4fd5e4dc-openshift-service-ca" pod "c024192c-6410-4363-89b9-4b0e4fd5e4dc" (UID: "c024192c-6410-4363-89b9-4b0e4fd5e4dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:59.001779 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:59.001735    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c024192c-6410-4363-89b9-4b0e4fd5e4dc-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:59.003317 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:59.003289    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c024192c-6410-4363-89b9-4b0e4fd5e4dc-maas-api-tls" pod "c024192c-6410-4363-89b9-4b0e4fd5e4dc" (UID: "c024192c-6410-4363-89b9-4b0e4fd5e4dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:59.003413 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:59.003360    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c024192c-6410-4363-89b9-4b0e4fd5e4dc-metrics-certs" pod "c024192c-6410-4363-89b9-4b0e4fd5e4dc" (UID: "c024192c-6410-4363-89b9-4b0e4fd5e4dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:59.003413 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:59.003376    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c024192c-6410-4363-89b9-4b0e4fd5e4dc-kube-api-access-rn42q" pod "c024192c-6410-4363-89b9-4b0e4fd5e4dc" (UID: "c024192c-6410-4363-89b9-4b0e4fd5e4dc"). InnerVolumeSpecName "kube-api-access-rn42q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:59.106516 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:59.103399    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c024192c-6410-4363-89b9-4b0e4fd5e4dc-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:59.106516 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:59.103432    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn42q\" (UniqueName: \"kubernetes.io/projected/c024192c-6410-4363-89b9-4b0e4fd5e4dc-kube-api-access-rn42q\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:59.106516 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:59.103447    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c024192c-6410-4363-89b9-4b0e4fd5e4dc-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:59.231340 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:59.231309    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"]
Oct 30 20:11:59.237379 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:59.237355    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d5e3e5-b6dc5db7d-7hfv8"]
Oct 30 20:11:59.540764 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:59.540721    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c024192c-6410-4363-89b9-4b0e4fd5e4dc" path="/var/lib/kubelet/pods/c024192c-6410-4363-89b9-4b0e4fd5e4dc/volumes"
Oct 30 20:11:59.804604 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:59.804502    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p" podUID="39539848-97ad-40d8-8082-d27a15143388" containerName="payload-processing" containerID="cri-o://029abd593bf48e9381e4f0ca4707219c3638238d28dd49f2de6ac2482231c1d1" gracePeriod=25
Oct 30 20:11:59.833818 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:11:59.833762    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4" podUID="686b83f0-b8a4-4210-a66d-e92c30e7fc8c" containerName="payload-pre-processing" containerID="cri-o://15b2c4ed74764dc5419c6a673ef511d8b24cbe5b8de5a81b4ffdeed3511b56e2" gracePeriod=25
Oct 30 20:12:00.111365 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.111333    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p"
Oct 30 20:12:00.135751 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.135726    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4"
Oct 30 20:12:00.272617 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.272576    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39539848-97ad-40d8-8082-d27a15143388-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39539848-97ad-40d8-8082-d27a15143388-plugins-config-volume\") pod \"39539848-97ad-40d8-8082-d27a15143388\" (UID: \"39539848-97ad-40d8-8082-d27a15143388\") "
Oct 30 20:12:00.272802 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.272785    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39539848-97ad-40d8-8082-d27a15143388-kube-api-access-twrxp\" (UniqueName: \"kubernetes.io/projected/39539848-97ad-40d8-8082-d27a15143388-kube-api-access-twrxp\") pod \"39539848-97ad-40d8-8082-d27a15143388\" (UID: \"39539848-97ad-40d8-8082-d27a15143388\") "
Oct 30 20:12:00.272917 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.272813    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/686b83f0-b8a4-4210-a66d-e92c30e7fc8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/686b83f0-b8a4-4210-a66d-e92c30e7fc8c-plugins-config-volume\") pod \"686b83f0-b8a4-4210-a66d-e92c30e7fc8c\" (UID: \"686b83f0-b8a4-4210-a66d-e92c30e7fc8c\") "
Oct 30 20:12:00.272917 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.272847    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/686b83f0-b8a4-4210-a66d-e92c30e7fc8c-kube-api-access-mjns7\" (UniqueName: \"kubernetes.io/projected/686b83f0-b8a4-4210-a66d-e92c30e7fc8c-kube-api-access-mjns7\") pod \"686b83f0-b8a4-4210-a66d-e92c30e7fc8c\" (UID: \"686b83f0-b8a4-4210-a66d-e92c30e7fc8c\") "
Oct 30 20:12:00.273334 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.273195    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39539848-97ad-40d8-8082-d27a15143388-plugins-config-volume" pod "39539848-97ad-40d8-8082-d27a15143388" (UID: "39539848-97ad-40d8-8082-d27a15143388"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:00.273557 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.273530    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686b83f0-b8a4-4210-a66d-e92c30e7fc8c-plugins-config-volume" pod "686b83f0-b8a4-4210-a66d-e92c30e7fc8c" (UID: "686b83f0-b8a4-4210-a66d-e92c30e7fc8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:00.273639 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.273602    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39539848-97ad-40d8-8082-d27a15143388-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:00.276009 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.275958    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39539848-97ad-40d8-8082-d27a15143388-kube-api-access-twrxp" pod "39539848-97ad-40d8-8082-d27a15143388" (UID: "39539848-97ad-40d8-8082-d27a15143388"). InnerVolumeSpecName "kube-api-access-twrxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:00.276397 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.276358    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686b83f0-b8a4-4210-a66d-e92c30e7fc8c-kube-api-access-mjns7" pod "686b83f0-b8a4-4210-a66d-e92c30e7fc8c" (UID: "686b83f0-b8a4-4210-a66d-e92c30e7fc8c"). InnerVolumeSpecName "kube-api-access-mjns7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:00.375444 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.375368    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twrxp\" (UniqueName: \"kubernetes.io/projected/39539848-97ad-40d8-8082-d27a15143388-kube-api-access-twrxp\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:00.375444 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.375398    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/686b83f0-b8a4-4210-a66d-e92c30e7fc8c-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:00.375444 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.375408    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjns7\" (UniqueName: \"kubernetes.io/projected/686b83f0-b8a4-4210-a66d-e92c30e7fc8c-kube-api-access-mjns7\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:00.918406 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.918369    5408 generic.go:356] "Generic (PLEG): container finished" podID="39539848-97ad-40d8-8082-d27a15143388" containerID="029abd593bf48e9381e4f0ca4707219c3638238d28dd49f2de6ac2482231c1d1" exitCode=0
Oct 30 20:12:00.918831 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.918424    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p"
Oct 30 20:12:00.918831 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.918462    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p" event={"ID":"39539848-97ad-40d8-8082-d27a15143388","Type":"ContainerDied","Data":"029abd593bf48e9381e4f0ca4707219c3638238d28dd49f2de6ac2482231c1d1"}
Oct 30 20:12:00.918831 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.918518    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p" event={"ID":"39539848-97ad-40d8-8082-d27a15143388","Type":"ContainerDied","Data":"ebbb8460ee9e3510ac9f176bba1d990e0f5598bfd71aba1b96a5dc06a34119e1"}
Oct 30 20:12:00.918831 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.918546    5408 scope.go:122] "RemoveContainer" containerID="029abd593bf48e9381e4f0ca4707219c3638238d28dd49f2de6ac2482231c1d1"
Oct 30 20:12:00.920147 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.920124    5408 generic.go:356] "Generic (PLEG): container finished" podID="686b83f0-b8a4-4210-a66d-e92c30e7fc8c" containerID="15b2c4ed74764dc5419c6a673ef511d8b24cbe5b8de5a81b4ffdeed3511b56e2" exitCode=0
Oct 30 20:12:00.920236 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.920161    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4" event={"ID":"686b83f0-b8a4-4210-a66d-e92c30e7fc8c","Type":"ContainerDied","Data":"15b2c4ed74764dc5419c6a673ef511d8b24cbe5b8de5a81b4ffdeed3511b56e2"}
Oct 30 20:12:00.920236 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.920173    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4"
Oct 30 20:12:00.920236 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.920200    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4" event={"ID":"686b83f0-b8a4-4210-a66d-e92c30e7fc8c","Type":"ContainerDied","Data":"5d45f524d27412c86c57b73403e6081f0d954478beacab7899bb2adbc80a2a8a"}
Oct 30 20:12:00.931098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.931080    5408 scope.go:122] "RemoveContainer" containerID="029abd593bf48e9381e4f0ca4707219c3638238d28dd49f2de6ac2482231c1d1"
Oct 30 20:12:00.931401 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:00.931375    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"029abd593bf48e9381e4f0ca4707219c3638238d28dd49f2de6ac2482231c1d1\": container with ID starting with 029abd593bf48e9381e4f0ca4707219c3638238d28dd49f2de6ac2482231c1d1 not found: ID does not exist" containerID="029abd593bf48e9381e4f0ca4707219c3638238d28dd49f2de6ac2482231c1d1"
Oct 30 20:12:00.931503 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.931411    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"029abd593bf48e9381e4f0ca4707219c3638238d28dd49f2de6ac2482231c1d1"} err="failed to get container status \"029abd593bf48e9381e4f0ca4707219c3638238d28dd49f2de6ac2482231c1d1\": rpc error: code = NotFound desc = could not find container \"029abd593bf48e9381e4f0ca4707219c3638238d28dd49f2de6ac2482231c1d1\": container with ID starting with 029abd593bf48e9381e4f0ca4707219c3638238d28dd49f2de6ac2482231c1d1 not found: ID does not exist"
Oct 30 20:12:00.931503 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.931434    5408 scope.go:122] "RemoveContainer" containerID="15b2c4ed74764dc5419c6a673ef511d8b24cbe5b8de5a81b4ffdeed3511b56e2"
Oct 30 20:12:00.941088 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.941072    5408 scope.go:122] "RemoveContainer" containerID="15b2c4ed74764dc5419c6a673ef511d8b24cbe5b8de5a81b4ffdeed3511b56e2"
Oct 30 20:12:00.941949 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:00.941931    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b2c4ed74764dc5419c6a673ef511d8b24cbe5b8de5a81b4ffdeed3511b56e2\": container with ID starting with 15b2c4ed74764dc5419c6a673ef511d8b24cbe5b8de5a81b4ffdeed3511b56e2 not found: ID does not exist" containerID="15b2c4ed74764dc5419c6a673ef511d8b24cbe5b8de5a81b4ffdeed3511b56e2"
Oct 30 20:12:00.942055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.941956    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b2c4ed74764dc5419c6a673ef511d8b24cbe5b8de5a81b4ffdeed3511b56e2"} err="failed to get container status \"15b2c4ed74764dc5419c6a673ef511d8b24cbe5b8de5a81b4ffdeed3511b56e2\": rpc error: code = NotFound desc = could not find container \"15b2c4ed74764dc5419c6a673ef511d8b24cbe5b8de5a81b4ffdeed3511b56e2\": container with ID starting with 15b2c4ed74764dc5419c6a673ef511d8b24cbe5b8de5a81b4ffdeed3511b56e2 not found: ID does not exist"
Oct 30 20:12:00.960591 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.960568    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4"]
Oct 30 20:12:00.975261 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:00.975228    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a8de3b-79c6c587-ttpd4"]
Oct 30 20:12:01.000220 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:01.000191    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p"]
Oct 30 20:12:01.003302 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:01.003277    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a8de3b-65dd7cc5c8-p7n7p"]
Oct 30 20:12:01.536920 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:01.536891    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39539848-97ad-40d8-8082-d27a15143388" path="/var/lib/kubelet/pods/39539848-97ad-40d8-8082-d27a15143388/volumes"
Oct 30 20:12:01.537450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:01.537427    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="686b83f0-b8a4-4210-a66d-e92c30e7fc8c" path="/var/lib/kubelet/pods/686b83f0-b8a4-4210-a66d-e92c30e7fc8c/volumes"
Oct 30 20:12:02.100997 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.100954    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7" podUID="cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1" containerName="payload-processing" containerID="cri-o://8ae0505b97d298038d12aef563c5fe18a53e7b37babbdfd95cc1e2e949a41cab" gracePeriod=25
Oct 30 20:12:02.117875 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.117837    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l" podUID="e48c3cc5-30d3-4bc8-98a7-548d78595ac6" containerName="payload-pre-processing" containerID="cri-o://3ee741d71f14112e10861253b06cc3a19ed55dd2e6b1fe5d22a0ab083c3c4a60" gracePeriod=25
Oct 30 20:12:02.417829 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.417799    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7"
Oct 30 20:12:02.497497 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.497468    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l"
Oct 30 20:12:02.528352 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.528319    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1-plugins-config-volume\") pod \"cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1\" (UID: \"cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1\") "
Oct 30 20:12:02.528520 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.528391    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1-kube-api-access-5ggzq\" (UniqueName: \"kubernetes.io/projected/cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1-kube-api-access-5ggzq\") pod \"cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1\" (UID: \"cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1\") "
Oct 30 20:12:02.530226 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.529821    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1-plugins-config-volume" pod "cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1" (UID: "cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:02.532677 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.532653    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1-kube-api-access-5ggzq" pod "cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1" (UID: "cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1"). InnerVolumeSpecName "kube-api-access-5ggzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:02.631410 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.630282    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e48c3cc5-30d3-4bc8-98a7-548d78595ac6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48c3cc5-30d3-4bc8-98a7-548d78595ac6-plugins-config-volume\") pod \"e48c3cc5-30d3-4bc8-98a7-548d78595ac6\" (UID: \"e48c3cc5-30d3-4bc8-98a7-548d78595ac6\") "
Oct 30 20:12:02.631410 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.630427    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e48c3cc5-30d3-4bc8-98a7-548d78595ac6-kube-api-access-cl9dx\" (UniqueName: \"kubernetes.io/projected/e48c3cc5-30d3-4bc8-98a7-548d78595ac6-kube-api-access-cl9dx\") pod \"e48c3cc5-30d3-4bc8-98a7-548d78595ac6\" (UID: \"e48c3cc5-30d3-4bc8-98a7-548d78595ac6\") "
Oct 30 20:12:02.633300 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.633271    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e48c3cc5-30d3-4bc8-98a7-548d78595ac6-plugins-config-volume" pod "e48c3cc5-30d3-4bc8-98a7-548d78595ac6" (UID: "e48c3cc5-30d3-4bc8-98a7-548d78595ac6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:02.633572 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.633553    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:02.633751 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.633719    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ggzq\" (UniqueName: \"kubernetes.io/projected/cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1-kube-api-access-5ggzq\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:02.645095 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.638221    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48c3cc5-30d3-4bc8-98a7-548d78595ac6-kube-api-access-cl9dx" pod "e48c3cc5-30d3-4bc8-98a7-548d78595ac6" (UID: "e48c3cc5-30d3-4bc8-98a7-548d78595ac6"). InnerVolumeSpecName "kube-api-access-cl9dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:02.734990 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.734958    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl9dx\" (UniqueName: \"kubernetes.io/projected/e48c3cc5-30d3-4bc8-98a7-548d78595ac6-kube-api-access-cl9dx\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:02.734990 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.734989    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48c3cc5-30d3-4bc8-98a7-548d78595ac6-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:02.939079 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.938948    5408 generic.go:356] "Generic (PLEG): container finished" podID="e48c3cc5-30d3-4bc8-98a7-548d78595ac6" containerID="3ee741d71f14112e10861253b06cc3a19ed55dd2e6b1fe5d22a0ab083c3c4a60" exitCode=0
Oct 30 20:12:02.939079 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.939013    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l"
Oct 30 20:12:02.939306 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.939007    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l" event={"ID":"e48c3cc5-30d3-4bc8-98a7-548d78595ac6","Type":"ContainerDied","Data":"3ee741d71f14112e10861253b06cc3a19ed55dd2e6b1fe5d22a0ab083c3c4a60"}
Oct 30 20:12:02.939306 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.939209    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l" event={"ID":"e48c3cc5-30d3-4bc8-98a7-548d78595ac6","Type":"ContainerDied","Data":"3cbfcbbb429efca202cad16971fb16e5571b9d2ccb46a69b9c267ebab079108d"}
Oct 30 20:12:02.939306 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.939246    5408 scope.go:122] "RemoveContainer" containerID="3ee741d71f14112e10861253b06cc3a19ed55dd2e6b1fe5d22a0ab083c3c4a60"
Oct 30 20:12:02.941188 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.940874    5408 generic.go:356] "Generic (PLEG): container finished" podID="cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1" containerID="8ae0505b97d298038d12aef563c5fe18a53e7b37babbdfd95cc1e2e949a41cab" exitCode=0
Oct 30 20:12:02.941188 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.940961    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7" event={"ID":"cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1","Type":"ContainerDied","Data":"8ae0505b97d298038d12aef563c5fe18a53e7b37babbdfd95cc1e2e949a41cab"}
Oct 30 20:12:02.941188 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.940994    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7" event={"ID":"cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1","Type":"ContainerDied","Data":"4ad41ba893b59450afdc16d164480a8bacdc011c7c58ce6ce527d7abfc151a26"}
Oct 30 20:12:02.941188 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.941082    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7"
Oct 30 20:12:03.022051 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.998597    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main" containerID="cri-o://84846debeba4483c62fe23687f46f9462f043eabeddb23f07df2c7fe6cabc40e" gracePeriod=45
Oct 30 20:12:03.022540 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:02.999931    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" containerID="cri-o://4299d2ed3393872930a03174f345fe1512fa6c322d77e02d128d40ed01a927f2" gracePeriod=45
Oct 30 20:12:03.022779 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.000861    5408 scope.go:122] "RemoveContainer" containerID="3ee741d71f14112e10861253b06cc3a19ed55dd2e6b1fe5d22a0ab083c3c4a60"
Oct 30 20:12:03.044594 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.044448    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7"]
Oct 30 20:12:03.067970 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:03.067295    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee741d71f14112e10861253b06cc3a19ed55dd2e6b1fe5d22a0ab083c3c4a60\": container with ID starting with 3ee741d71f14112e10861253b06cc3a19ed55dd2e6b1fe5d22a0ab083c3c4a60 not found: ID does not exist" containerID="3ee741d71f14112e10861253b06cc3a19ed55dd2e6b1fe5d22a0ab083c3c4a60"
Oct 30 20:12:03.067970 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.067343    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee741d71f14112e10861253b06cc3a19ed55dd2e6b1fe5d22a0ab083c3c4a60"} err="failed to get container status \"3ee741d71f14112e10861253b06cc3a19ed55dd2e6b1fe5d22a0ab083c3c4a60\": rpc error: code = NotFound desc = could not find container \"3ee741d71f14112e10861253b06cc3a19ed55dd2e6b1fe5d22a0ab083c3c4a60\": container with ID starting with 3ee741d71f14112e10861253b06cc3a19ed55dd2e6b1fe5d22a0ab083c3c4a60 not found: ID does not exist"
Oct 30 20:12:03.067970 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.067370    5408 scope.go:122] "RemoveContainer" containerID="8ae0505b97d298038d12aef563c5fe18a53e7b37babbdfd95cc1e2e949a41cab"
Oct 30 20:12:03.080462 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.079152    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d7a733-65c7dd8844-mrpm7"]
Oct 30 20:12:03.178439 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.178408    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l"]
Oct 30 20:12:03.179754 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.179730    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d7a733-5c4448594c-c8p8l"]
Oct 30 20:12:03.180014 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.180001    5408 scope.go:122] "RemoveContainer" containerID="8ae0505b97d298038d12aef563c5fe18a53e7b37babbdfd95cc1e2e949a41cab"
Oct 30 20:12:03.188183 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:03.187936    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae0505b97d298038d12aef563c5fe18a53e7b37babbdfd95cc1e2e949a41cab\": container with ID starting with 8ae0505b97d298038d12aef563c5fe18a53e7b37babbdfd95cc1e2e949a41cab not found: ID does not exist" containerID="8ae0505b97d298038d12aef563c5fe18a53e7b37babbdfd95cc1e2e949a41cab"
Oct 30 20:12:03.188718 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.188443    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": dial tcp 10.134.0.58:8000: connect: connection refused"
Oct 30 20:12:03.188718 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.188616    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae0505b97d298038d12aef563c5fe18a53e7b37babbdfd95cc1e2e949a41cab"} err="failed to get container status \"8ae0505b97d298038d12aef563c5fe18a53e7b37babbdfd95cc1e2e949a41cab\": rpc error: code = NotFound desc = could not find container \"8ae0505b97d298038d12aef563c5fe18a53e7b37babbdfd95cc1e2e949a41cab\": container with ID starting with 8ae0505b97d298038d12aef563c5fe18a53e7b37babbdfd95cc1e2e949a41cab not found: ID does not exist"
Oct 30 20:12:03.385487 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.385464    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 20:12:03.433841 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.433811    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 20:12:03.485283 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.485202    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-kube-api-access-6sd2l\" (UniqueName: \"kubernetes.io/projected/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-kube-api-access-6sd2l\") pod \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") "
Oct 30 20:12:03.485283 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.485250    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f141028a-4c85-40ca-b0aa-cdc0838170d1-kube-api-access-qvmrz\" (UniqueName: \"kubernetes.io/projected/f141028a-4c85-40ca-b0aa-cdc0838170d1-kube-api-access-qvmrz\") pod \"f141028a-4c85-40ca-b0aa-cdc0838170d1\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") "
Oct 30 20:12:03.485514 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.485300    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-tmp-dir\") pod \"f141028a-4c85-40ca-b0aa-cdc0838170d1\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") "
Oct 30 20:12:03.485514 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.485346    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-model-cache\") pod \"f141028a-4c85-40ca-b0aa-cdc0838170d1\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") "
Oct 30 20:12:03.485514 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.485407    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-dshm\") pod \"f141028a-4c85-40ca-b0aa-cdc0838170d1\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") "
Oct 30 20:12:03.485514 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.485434    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-tls-certs\") pod \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") "
Oct 30 20:12:03.485728 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.485541    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-tmp-dir\") pod \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") "
Oct 30 20:12:03.485728 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.485561    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-tmp-dir" pod "f141028a-4c85-40ca-b0aa-cdc0838170d1" (UID: "f141028a-4c85-40ca-b0aa-cdc0838170d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:03.485728 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.485579    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-model-cache\") pod \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") "
Oct 30 20:12:03.485728 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.485642    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-dshm\") pod \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") "
Oct 30 20:12:03.485942 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.485778    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-home\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-home\") pod \"f141028a-4c85-40ca-b0aa-cdc0838170d1\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") "
Oct 30 20:12:03.485942 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.485821    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-home\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-home\") pod \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\" (UID: \"6a886ba9-fe5e-4739-bb26-7bdb235e6e64\") "
Oct 30 20:12:03.485942 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.485875    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f141028a-4c85-40ca-b0aa-cdc0838170d1-tls-certs\" (UniqueName: \"kubernetes.io/secret/f141028a-4c85-40ca-b0aa-cdc0838170d1-tls-certs\") pod \"f141028a-4c85-40ca-b0aa-cdc0838170d1\" (UID: \"f141028a-4c85-40ca-b0aa-cdc0838170d1\") "
Oct 30 20:12:03.488172 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.486350    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-home" pod "f141028a-4c85-40ca-b0aa-cdc0838170d1" (UID: "f141028a-4c85-40ca-b0aa-cdc0838170d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:03.488172 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.486581    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-model-cache" pod "f141028a-4c85-40ca-b0aa-cdc0838170d1" (UID: "f141028a-4c85-40ca-b0aa-cdc0838170d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:03.488172 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.487826    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-tmp-dir" pod "6a886ba9-fe5e-4739-bb26-7bdb235e6e64" (UID: "6a886ba9-fe5e-4739-bb26-7bdb235e6e64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:03.488172 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.487858    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-home" pod "6a886ba9-fe5e-4739-bb26-7bdb235e6e64" (UID: "6a886ba9-fe5e-4739-bb26-7bdb235e6e64"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:03.488172 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.488079    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:03.488172 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.488104    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:03.488172 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.488121    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:03.488172 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.488134    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:03.488172 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.488146    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:03.488172 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.488158    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-dshm" pod "f141028a-4c85-40ca-b0aa-cdc0838170d1" (UID: "f141028a-4c85-40ca-b0aa-cdc0838170d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:03.488750 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.488587    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f141028a-4c85-40ca-b0aa-cdc0838170d1-kube-api-access-qvmrz" pod "f141028a-4c85-40ca-b0aa-cdc0838170d1" (UID: "f141028a-4c85-40ca-b0aa-cdc0838170d1"). InnerVolumeSpecName "kube-api-access-qvmrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:03.488919 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.488893    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-model-cache" pod "6a886ba9-fe5e-4739-bb26-7bdb235e6e64" (UID: "6a886ba9-fe5e-4739-bb26-7bdb235e6e64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:03.489236 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.489210    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-tls-certs" pod "6a886ba9-fe5e-4739-bb26-7bdb235e6e64" (UID: "6a886ba9-fe5e-4739-bb26-7bdb235e6e64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:03.489822 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.489792    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-kube-api-access-6sd2l" pod "6a886ba9-fe5e-4739-bb26-7bdb235e6e64" (UID: "6a886ba9-fe5e-4739-bb26-7bdb235e6e64"). InnerVolumeSpecName "kube-api-access-6sd2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:03.490592 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.490566    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f141028a-4c85-40ca-b0aa-cdc0838170d1-tls-certs" pod "f141028a-4c85-40ca-b0aa-cdc0838170d1" (UID: "f141028a-4c85-40ca-b0aa-cdc0838170d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:03.491316 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.491295    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-dshm" pod "6a886ba9-fe5e-4739-bb26-7bdb235e6e64" (UID: "6a886ba9-fe5e-4739-bb26-7bdb235e6e64"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:03.543434 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.543402    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1" path="/var/lib/kubelet/pods/cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1/volumes"
Oct 30 20:12:03.543911 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.543894    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e48c3cc5-30d3-4bc8-98a7-548d78595ac6" path="/var/lib/kubelet/pods/e48c3cc5-30d3-4bc8-98a7-548d78595ac6/volumes"
Oct 30 20:12:03.589639 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.589610    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sd2l\" (UniqueName: \"kubernetes.io/projected/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-kube-api-access-6sd2l\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:03.589639 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.589637    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvmrz\" (UniqueName: \"kubernetes.io/projected/f141028a-4c85-40ca-b0aa-cdc0838170d1-kube-api-access-qvmrz\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:03.589639 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.589648    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f141028a-4c85-40ca-b0aa-cdc0838170d1-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:03.589866 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.589657    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:03.589866 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.589665    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:03.589866 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.589673    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a886ba9-fe5e-4739-bb26-7bdb235e6e64-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:03.589866 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.589687    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f141028a-4c85-40ca-b0aa-cdc0838170d1-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:03.956214 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.956166    5408 generic.go:356] "Generic (PLEG): container finished" podID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerID="4299d2ed3393872930a03174f345fe1512fa6c322d77e02d128d40ed01a927f2" exitCode=0
Oct 30 20:12:03.956489 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.956291    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" event={"ID":"6a886ba9-fe5e-4739-bb26-7bdb235e6e64","Type":"ContainerDied","Data":"4299d2ed3393872930a03174f345fe1512fa6c322d77e02d128d40ed01a927f2"}
Oct 30 20:12:03.956489 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.956339    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk" event={"ID":"6a886ba9-fe5e-4739-bb26-7bdb235e6e64","Type":"ContainerDied","Data":"74d8e1fad8227b8d3db82adfe859a613ee335428edab0d9b972201a7eab52d76"}
Oct 30 20:12:03.956489 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.956364    5408 scope.go:122] "RemoveContainer" containerID="4299d2ed3393872930a03174f345fe1512fa6c322d77e02d128d40ed01a927f2"
Oct 30 20:12:03.959014 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.958846    5408 generic.go:356] "Generic (PLEG): container finished" podID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerID="84846debeba4483c62fe23687f46f9462f043eabeddb23f07df2c7fe6cabc40e" exitCode=0
Oct 30 20:12:03.959014 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.958950    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"
Oct 30 20:12:03.959014 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.958959    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" event={"ID":"f141028a-4c85-40ca-b0aa-cdc0838170d1","Type":"ContainerDied","Data":"84846debeba4483c62fe23687f46f9462f043eabeddb23f07df2c7fe6cabc40e"}
Oct 30 20:12:03.959014 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.958989    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t" event={"ID":"f141028a-4c85-40ca-b0aa-cdc0838170d1","Type":"ContainerDied","Data":"0837f479a2f75570d54ffa23901feebedc0060a046008a04c0932b53eea102f9"}
Oct 30 20:12:03.970467 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.970435    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"
Oct 30 20:12:03.970726 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.970702    5408 scope.go:122] "RemoveContainer" containerID="4299d2ed3393872930a03174f345fe1512fa6c322d77e02d128d40ed01a927f2"
Oct 30 20:12:03.971163 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:03.971136    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4299d2ed3393872930a03174f345fe1512fa6c322d77e02d128d40ed01a927f2\": container with ID starting with 4299d2ed3393872930a03174f345fe1512fa6c322d77e02d128d40ed01a927f2 not found: ID does not exist" containerID="4299d2ed3393872930a03174f345fe1512fa6c322d77e02d128d40ed01a927f2"
Oct 30 20:12:03.971263 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.971173    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4299d2ed3393872930a03174f345fe1512fa6c322d77e02d128d40ed01a927f2"} err="failed to get container status \"4299d2ed3393872930a03174f345fe1512fa6c322d77e02d128d40ed01a927f2\": rpc error: code = NotFound desc = could not find container \"4299d2ed3393872930a03174f345fe1512fa6c322d77e02d128d40ed01a927f2\": container with ID starting with 4299d2ed3393872930a03174f345fe1512fa6c322d77e02d128d40ed01a927f2 not found: ID does not exist"
Oct 30 20:12:03.971263 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.971198    5408 scope.go:122] "RemoveContainer" containerID="84846debeba4483c62fe23687f46f9462f043eabeddb23f07df2c7fe6cabc40e"
Oct 30 20:12:03.984206 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.984059    5408 scope.go:122] "RemoveContainer" containerID="84846debeba4483c62fe23687f46f9462f043eabeddb23f07df2c7fe6cabc40e"
Oct 30 20:12:03.984464 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:03.984435    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84846debeba4483c62fe23687f46f9462f043eabeddb23f07df2c7fe6cabc40e\": container with ID starting with 84846debeba4483c62fe23687f46f9462f043eabeddb23f07df2c7fe6cabc40e not found: ID does not exist" containerID="84846debeba4483c62fe23687f46f9462f043eabeddb23f07df2c7fe6cabc40e"
Oct 30 20:12:03.984555 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:03.984470    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84846debeba4483c62fe23687f46f9462f043eabeddb23f07df2c7fe6cabc40e"} err="failed to get container status \"84846debeba4483c62fe23687f46f9462f043eabeddb23f07df2c7fe6cabc40e\": rpc error: code = NotFound desc = could not find container \"84846debeba4483c62fe23687f46f9462f043eabeddb23f07df2c7fe6cabc40e\": container with ID starting with 84846debeba4483c62fe23687f46f9462f043eabeddb23f07df2c7fe6cabc40e not found: ID does not exist"
Oct 30 20:12:04.019576 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:04.019547    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"]
Oct 30 20:12:04.026170 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:04.026140    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/premium-w1-a1d885-kserve-5b494dbc9d-bjj9t"]
Oct 30 20:12:04.056457 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:04.056419    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"]
Oct 30 20:12:04.062612 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:04.062576    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/distinct-w1-a1d885-kserve-579f467465-kgvrk"]
Oct 30 20:12:05.469078 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:05.469043    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"]
Oct 30 20:12:05.469948 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:05.469911    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc" podUID="3fc2d2e5-3aab-4f01-997b-07aa7841dd73" containerName="maas-api" containerID="cri-o://d46cc2f75c49f61cf5c9a62bcb8c58b3f142fb3089388e48d2334789f92db9d0" gracePeriod=30
Oct 30 20:12:05.535097 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:05.535068    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" path="/var/lib/kubelet/pods/6a886ba9-fe5e-4739-bb26-7bdb235e6e64/volumes"
Oct 30 20:12:05.535546 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:05.535532    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" path="/var/lib/kubelet/pods/f141028a-4c85-40ca-b0aa-cdc0838170d1/volumes"
Oct 30 20:12:06.027377 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.027095    5408 generic.go:356] "Generic (PLEG): container finished" podID="3fc2d2e5-3aab-4f01-997b-07aa7841dd73" containerID="d46cc2f75c49f61cf5c9a62bcb8c58b3f142fb3089388e48d2334789f92db9d0" exitCode=0
Oct 30 20:12:06.027377 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.027281    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc" event={"ID":"3fc2d2e5-3aab-4f01-997b-07aa7841dd73","Type":"ContainerDied","Data":"d46cc2f75c49f61cf5c9a62bcb8c58b3f142fb3089388e48d2334789f92db9d0"}
Oct 30 20:12:06.027377 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.027329    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc" event={"ID":"3fc2d2e5-3aab-4f01-997b-07aa7841dd73","Type":"ContainerDied","Data":"ee96bd98ad0a3ac5fe09e93861ef8a80460662b7d87dedbd0522552b7a786ae0"}
Oct 30 20:12:06.027377 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.027352    5408 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee96bd98ad0a3ac5fe09e93861ef8a80460662b7d87dedbd0522552b7a786ae0"
Oct 30 20:12:06.029447 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.029426    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"
Oct 30 20:12:06.100680 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.100648    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-kube-api-access-nwvx7\" (UniqueName: \"kubernetes.io/projected/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-kube-api-access-nwvx7\") pod \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\" (UID: \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\") "
Oct 30 20:12:06.100856 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.100723    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-metrics-certs\") pod \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\" (UID: \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\") "
Oct 30 20:12:06.100856 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.100822    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-openshift-service-ca\") pod \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\" (UID: \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\") "
Oct 30 20:12:06.100973 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.100898    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-maas-api-tls\") pod \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\" (UID: \"3fc2d2e5-3aab-4f01-997b-07aa7841dd73\") "
Oct 30 20:12:06.101277 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.101253    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-openshift-service-ca" pod "3fc2d2e5-3aab-4f01-997b-07aa7841dd73" (UID: "3fc2d2e5-3aab-4f01-997b-07aa7841dd73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:06.103440 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.103417    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-metrics-certs" pod "3fc2d2e5-3aab-4f01-997b-07aa7841dd73" (UID: "3fc2d2e5-3aab-4f01-997b-07aa7841dd73"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:06.103581 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.103470    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-maas-api-tls" pod "3fc2d2e5-3aab-4f01-997b-07aa7841dd73" (UID: "3fc2d2e5-3aab-4f01-997b-07aa7841dd73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:06.103581 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.103494    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-kube-api-access-nwvx7" pod "3fc2d2e5-3aab-4f01-997b-07aa7841dd73" (UID: "3fc2d2e5-3aab-4f01-997b-07aa7841dd73"). InnerVolumeSpecName "kube-api-access-nwvx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:06.202937 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.202850    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:06.202937 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.202879    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:06.202937 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.202894    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwvx7\" (UniqueName: \"kubernetes.io/projected/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-kube-api-access-nwvx7\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:06.202937 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.202907    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fc2d2e5-3aab-4f01-997b-07aa7841dd73-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:06.235870 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.235837    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"]
Oct 30 20:12:06.236157 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:06.236131    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj" podUID="e0c23612-28be-4e5f-aa95-71c40e82450e" containerName="istio-proxy" containerID="cri-o://08f8819f856fcff97a48ed9f69499641fc8669ca4125379ee6e269bdae2f3ac0" gracePeriod=30
Oct 30 20:12:07.034543 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:07.034510    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"
Oct 30 20:12:07.072963 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:07.072934    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"]
Oct 30 20:12:07.076484 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:07.076464    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-82885f-7f647dc646-qhwfc"]
Oct 30 20:12:07.532976 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:07.532941    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc2d2e5-3aab-4f01-997b-07aa7841dd73" path="/var/lib/kubelet/pods/3fc2d2e5-3aab-4f01-997b-07aa7841dd73/volumes"
Oct 30 20:12:09.274398 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:09.274366    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"]
Oct 30 20:12:09.274930 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:09.274641    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq" podUID="688f0cb2-8d4a-47ab-ab79-fb1a54109e48" containerName="istio-proxy" containerID="cri-o://7d7805b2830bcc21f460e41c7be2ef41ba2d2357171e84e806496ca083d7a23c" gracePeriod=30
Oct 30 20:12:09.279685 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:09.279658    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"]
Oct 30 20:12:09.279927 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:09.279892    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy" containerID="cri-o://2185aaaf7d5bb4c253ea7068005c6a68475e80a78eea0c86bcd2121a1248a8f4" gracePeriod=30
Oct 30 20:12:09.293816 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:09.293782    5408 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:12:09.293926 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:09.293864    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj" podUID="e0c23612-28be-4e5f-aa95-71c40e82450e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:12:09.672522 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:09.672428    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:12:10.655304 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:10.655268    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"]
Oct 30 20:12:10.656085 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:10.656051    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk" podUID="d8c8ac1e-8cf9-4a86-affa-9a2745709749" containerName="maas-api" containerID="cri-o://1ad6f6cb2c0f9f5b2833fb6daccc0266b357a446399418b0600eb760b01167ca" gracePeriod=30
Oct 30 20:12:11.179243 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.179212    5408 generic.go:356] "Generic (PLEG): container finished" podID="d8c8ac1e-8cf9-4a86-affa-9a2745709749" containerID="1ad6f6cb2c0f9f5b2833fb6daccc0266b357a446399418b0600eb760b01167ca" exitCode=0
Oct 30 20:12:11.179567 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.179466    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk" event={"ID":"d8c8ac1e-8cf9-4a86-affa-9a2745709749","Type":"ContainerDied","Data":"1ad6f6cb2c0f9f5b2833fb6daccc0266b357a446399418b0600eb760b01167ca"}
Oct 30 20:12:11.232218 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.232187    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"
Oct 30 20:12:11.363332 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.363306    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c8ac1e-8cf9-4a86-affa-9a2745709749-kube-api-access-bmq8d\" (UniqueName: \"kubernetes.io/projected/d8c8ac1e-8cf9-4a86-affa-9a2745709749-kube-api-access-bmq8d\") pod \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\" (UID: \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\") "
Oct 30 20:12:11.363454 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.363436    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8c8ac1e-8cf9-4a86-affa-9a2745709749-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c8ac1e-8cf9-4a86-affa-9a2745709749-openshift-service-ca\") pod \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\" (UID: \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\") "
Oct 30 20:12:11.363621 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.363606    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c8ac1e-8cf9-4a86-affa-9a2745709749-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c8ac1e-8cf9-4a86-affa-9a2745709749-maas-api-tls\") pod \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\" (UID: \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\") "
Oct 30 20:12:11.363690 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.363678    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c8ac1e-8cf9-4a86-affa-9a2745709749-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c8ac1e-8cf9-4a86-affa-9a2745709749-metrics-certs\") pod \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\" (UID: \"d8c8ac1e-8cf9-4a86-affa-9a2745709749\") "
Oct 30 20:12:11.363793 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.363756    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c8ac1e-8cf9-4a86-affa-9a2745709749-openshift-service-ca" pod "d8c8ac1e-8cf9-4a86-affa-9a2745709749" (UID: "d8c8ac1e-8cf9-4a86-affa-9a2745709749"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:11.364382 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.364338    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c8ac1e-8cf9-4a86-affa-9a2745709749-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:11.365894 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.365865    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c8ac1e-8cf9-4a86-affa-9a2745709749-kube-api-access-bmq8d" pod "d8c8ac1e-8cf9-4a86-affa-9a2745709749" (UID: "d8c8ac1e-8cf9-4a86-affa-9a2745709749"). InnerVolumeSpecName "kube-api-access-bmq8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:11.365988 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.365920    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c8ac1e-8cf9-4a86-affa-9a2745709749-maas-api-tls" pod "d8c8ac1e-8cf9-4a86-affa-9a2745709749" (UID: "d8c8ac1e-8cf9-4a86-affa-9a2745709749"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:11.366349 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.366330    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c8ac1e-8cf9-4a86-affa-9a2745709749-metrics-certs" pod "d8c8ac1e-8cf9-4a86-affa-9a2745709749" (UID: "d8c8ac1e-8cf9-4a86-affa-9a2745709749"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:11.520222 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.471314    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c8ac1e-8cf9-4a86-affa-9a2745709749-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:11.520222 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.471354    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c8ac1e-8cf9-4a86-affa-9a2745709749-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:11.520222 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.471405    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmq8d\" (UniqueName: \"kubernetes.io/projected/d8c8ac1e-8cf9-4a86-affa-9a2745709749-kube-api-access-bmq8d\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:11.565481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.565448    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:12:11.674989 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.674951    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-data\") pod \"e0c23612-28be-4e5f-aa95-71c40e82450e\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") "
Oct 30 20:12:11.675495 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.675124    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c23612-28be-4e5f-aa95-71c40e82450e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0c23612-28be-4e5f-aa95-71c40e82450e-istiod-ca-cert\") pod \"e0c23612-28be-4e5f-aa95-71c40e82450e\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") "
Oct 30 20:12:11.675495 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.675160    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-credential-socket\") pod \"e0c23612-28be-4e5f-aa95-71c40e82450e\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") "
Oct 30 20:12:11.675495 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.675270    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c23612-28be-4e5f-aa95-71c40e82450e-kube-api-access-vw7b7\" (UniqueName: \"kubernetes.io/projected/e0c23612-28be-4e5f-aa95-71c40e82450e-kube-api-access-vw7b7\") pod \"e0c23612-28be-4e5f-aa95-71c40e82450e\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") "
Oct 30 20:12:11.675495 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.675346    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-data" pod "e0c23612-28be-4e5f-aa95-71c40e82450e" (UID: "e0c23612-28be-4e5f-aa95-71c40e82450e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:11.675495 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.675351    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-token\" (UniqueName: \"kubernetes.io/projected/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-token\") pod \"e0c23612-28be-4e5f-aa95-71c40e82450e\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") "
Oct 30 20:12:11.675495 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.675453    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-envoy\") pod \"e0c23612-28be-4e5f-aa95-71c40e82450e\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") "
Oct 30 20:12:11.675495 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.675479    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-credential-socket" pod "e0c23612-28be-4e5f-aa95-71c40e82450e" (UID: "e0c23612-28be-4e5f-aa95-71c40e82450e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:11.675887 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.675507    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-podinfo\") pod \"e0c23612-28be-4e5f-aa95-71c40e82450e\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") "
Oct 30 20:12:11.675887 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.675580    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-workload-certs\") pod \"e0c23612-28be-4e5f-aa95-71c40e82450e\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") "
Oct 30 20:12:11.675887 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.675634    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-workload-socket\") pod \"e0c23612-28be-4e5f-aa95-71c40e82450e\" (UID: \"e0c23612-28be-4e5f-aa95-71c40e82450e\") "
Oct 30 20:12:11.676060 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.675579    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c23612-28be-4e5f-aa95-71c40e82450e-istiod-ca-cert" pod "e0c23612-28be-4e5f-aa95-71c40e82450e" (UID: "e0c23612-28be-4e5f-aa95-71c40e82450e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:11.676060 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.675910    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-workload-certs" pod "e0c23612-28be-4e5f-aa95-71c40e82450e" (UID: "e0c23612-28be-4e5f-aa95-71c40e82450e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:11.676175 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.676159    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-workload-socket" pod "e0c23612-28be-4e5f-aa95-71c40e82450e" (UID: "e0c23612-28be-4e5f-aa95-71c40e82450e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:11.676391 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.676373    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:11.676489 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.676417    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0c23612-28be-4e5f-aa95-71c40e82450e-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:11.676489 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.676434    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:11.676489 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.676448    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:11.676489 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.676462    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:11.678094 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.678021    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-podinfo" pod "e0c23612-28be-4e5f-aa95-71c40e82450e" (UID: "e0c23612-28be-4e5f-aa95-71c40e82450e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:12:11.678094 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.678081    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-envoy" pod "e0c23612-28be-4e5f-aa95-71c40e82450e" (UID: "e0c23612-28be-4e5f-aa95-71c40e82450e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:11.678272 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.678104    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-token" pod "e0c23612-28be-4e5f-aa95-71c40e82450e" (UID: "e0c23612-28be-4e5f-aa95-71c40e82450e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:11.678321 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.678305    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c23612-28be-4e5f-aa95-71c40e82450e-kube-api-access-vw7b7" pod "e0c23612-28be-4e5f-aa95-71c40e82450e" (UID: "e0c23612-28be-4e5f-aa95-71c40e82450e"). InnerVolumeSpecName "kube-api-access-vw7b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:11.777806 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.777714    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw7b7\" (UniqueName: \"kubernetes.io/projected/e0c23612-28be-4e5f-aa95-71c40e82450e-kube-api-access-vw7b7\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:11.777806 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.777755    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:11.777806 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.777771    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:11.777806 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:11.777783    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0c23612-28be-4e5f-aa95-71c40e82450e-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:12.236425 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.236047    5408 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:12:12.236425 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.236110    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:12:12.264165 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.264129    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk" event={"ID":"d8c8ac1e-8cf9-4a86-affa-9a2745709749","Type":"ContainerDied","Data":"56d5399430f0894a9b47fd0699f3b498a18dedd372f50fa7f0967196cdc790da"}
Oct 30 20:12:12.264349 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.264202    5408 scope.go:122] "RemoveContainer" containerID="1ad6f6cb2c0f9f5b2833fb6daccc0266b357a446399418b0600eb760b01167ca"
Oct 30 20:12:12.264349 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.264347    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"
Oct 30 20:12:12.277318 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.270486    5408 generic.go:356] "Generic (PLEG): container finished" podID="e0c23612-28be-4e5f-aa95-71c40e82450e" containerID="08f8819f856fcff97a48ed9f69499641fc8669ca4125379ee6e269bdae2f3ac0" exitCode=0
Oct 30 20:12:12.277318 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.270686    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj" event={"ID":"e0c23612-28be-4e5f-aa95-71c40e82450e","Type":"ContainerDied","Data":"08f8819f856fcff97a48ed9f69499641fc8669ca4125379ee6e269bdae2f3ac0"}
Oct 30 20:12:12.277318 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.270733    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj" event={"ID":"e0c23612-28be-4e5f-aa95-71c40e82450e","Type":"ContainerDied","Data":"3e980333295940baea2db90b1c2dc8fd01f9b402e65ceb2db86500fa8874fecd"}
Oct 30 20:12:12.277318 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.270847    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"
Oct 30 20:12:12.430310 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.430229    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"]
Oct 30 20:12:12.440954 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.440739    5408 scope.go:122] "RemoveContainer" containerID="08f8819f856fcff97a48ed9f69499641fc8669ca4125379ee6e269bdae2f3ac0"
Oct 30 20:12:12.458125 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.443552    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a1d885-86b7d9d78f-2h8lk"]
Oct 30 20:12:12.458709 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.458638    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"]
Oct 30 20:12:12.503565 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.503484    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-wqlhj"]
Oct 30 20:12:12.537248 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.534955    5408 scope.go:122] "RemoveContainer" containerID="7a18ecc51e65dc731022f8c617b5d0794e8c7b58b5ca85e8e7962b5e2658d9e0"
Oct 30 20:12:12.758829 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.758668    5408 scope.go:122] "RemoveContainer" containerID="08f8819f856fcff97a48ed9f69499641fc8669ca4125379ee6e269bdae2f3ac0"
Oct 30 20:12:12.888815 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:12.888773    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f8819f856fcff97a48ed9f69499641fc8669ca4125379ee6e269bdae2f3ac0\": container with ID starting with 08f8819f856fcff97a48ed9f69499641fc8669ca4125379ee6e269bdae2f3ac0 not found: ID does not exist" containerID="08f8819f856fcff97a48ed9f69499641fc8669ca4125379ee6e269bdae2f3ac0"
Oct 30 20:12:12.889005 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.888830    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f8819f856fcff97a48ed9f69499641fc8669ca4125379ee6e269bdae2f3ac0"} err="failed to get container status \"08f8819f856fcff97a48ed9f69499641fc8669ca4125379ee6e269bdae2f3ac0\": rpc error: code = NotFound desc = could not find container \"08f8819f856fcff97a48ed9f69499641fc8669ca4125379ee6e269bdae2f3ac0\": container with ID starting with 08f8819f856fcff97a48ed9f69499641fc8669ca4125379ee6e269bdae2f3ac0 not found: ID does not exist"
Oct 30 20:12:12.889005 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.888856    5408 scope.go:122] "RemoveContainer" containerID="7a18ecc51e65dc731022f8c617b5d0794e8c7b58b5ca85e8e7962b5e2658d9e0"
Oct 30 20:12:12.889508 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:12.889467    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a18ecc51e65dc731022f8c617b5d0794e8c7b58b5ca85e8e7962b5e2658d9e0\": container with ID starting with 7a18ecc51e65dc731022f8c617b5d0794e8c7b58b5ca85e8e7962b5e2658d9e0 not found: ID does not exist" containerID="7a18ecc51e65dc731022f8c617b5d0794e8c7b58b5ca85e8e7962b5e2658d9e0"
Oct 30 20:12:12.889624 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:12.889532    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a18ecc51e65dc731022f8c617b5d0794e8c7b58b5ca85e8e7962b5e2658d9e0"} err="failed to get container status \"7a18ecc51e65dc731022f8c617b5d0794e8c7b58b5ca85e8e7962b5e2658d9e0\": rpc error: code = NotFound desc = could not find container \"7a18ecc51e65dc731022f8c617b5d0794e8c7b58b5ca85e8e7962b5e2658d9e0\": container with ID starting with 7a18ecc51e65dc731022f8c617b5d0794e8c7b58b5ca85e8e7962b5e2658d9e0 not found: ID does not exist"
Oct 30 20:12:13.533459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:13.533431    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c8ac1e-8cf9-4a86-affa-9a2745709749" path="/var/lib/kubelet/pods/d8c8ac1e-8cf9-4a86-affa-9a2745709749/volumes"
Oct 30 20:12:13.533952 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:13.533939    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c23612-28be-4e5f-aa95-71c40e82450e" path="/var/lib/kubelet/pods/e0c23612-28be-4e5f-aa95-71c40e82450e/volumes"
Oct 30 20:12:14.723168 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.722762    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 20:12:14.786455 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.786416    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-data\") pod \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") "
Oct 30 20:12:14.786617 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.786509    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istiod-ca-cert\") pod \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") "
Oct 30 20:12:14.786617 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.786601    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-podinfo\") pod \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") "
Oct 30 20:12:14.786698 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.786677    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-token\" (UniqueName: \"kubernetes.io/projected/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-token\") pod \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") "
Oct 30 20:12:14.786762 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.786747    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-kube-api-access-nkjc9\" (UniqueName: \"kubernetes.io/projected/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-kube-api-access-nkjc9\") pod \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") "
Oct 30 20:12:14.786821 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.786808    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-workload-certs\") pod \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") "
Oct 30 20:12:14.786864 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.786850    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-credential-socket\") pod \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") "
Oct 30 20:12:14.786918 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.786904    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-envoy\") pod \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") "
Oct 30 20:12:14.787065 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.787013    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-workload-socket\") pod \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\" (UID: \"688f0cb2-8d4a-47ab-ab79-fb1a54109e48\") "
Oct 30 20:12:14.787981 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.787955    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-workload-socket" pod "688f0cb2-8d4a-47ab-ab79-fb1a54109e48" (UID: "688f0cb2-8d4a-47ab-ab79-fb1a54109e48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:14.788132 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.788117    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-credential-socket" pod "688f0cb2-8d4a-47ab-ab79-fb1a54109e48" (UID: "688f0cb2-8d4a-47ab-ab79-fb1a54109e48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:14.789113 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.788965    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-data" pod "688f0cb2-8d4a-47ab-ab79-fb1a54109e48" (UID: "688f0cb2-8d4a-47ab-ab79-fb1a54109e48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:14.789113 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.789086    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-workload-certs" pod "688f0cb2-8d4a-47ab-ab79-fb1a54109e48" (UID: "688f0cb2-8d4a-47ab-ab79-fb1a54109e48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:14.791808 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.791761    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istiod-ca-cert" pod "688f0cb2-8d4a-47ab-ab79-fb1a54109e48" (UID: "688f0cb2-8d4a-47ab-ab79-fb1a54109e48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:14.797367 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.797333    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-podinfo" pod "688f0cb2-8d4a-47ab-ab79-fb1a54109e48" (UID: "688f0cb2-8d4a-47ab-ab79-fb1a54109e48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:12:14.798547 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.798318    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-kube-api-access-nkjc9" pod "688f0cb2-8d4a-47ab-ab79-fb1a54109e48" (UID: "688f0cb2-8d4a-47ab-ab79-fb1a54109e48"). InnerVolumeSpecName "kube-api-access-nkjc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:14.798547 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.798384    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-envoy" pod "688f0cb2-8d4a-47ab-ab79-fb1a54109e48" (UID: "688f0cb2-8d4a-47ab-ab79-fb1a54109e48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:14.816533 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.816451    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-token" pod "688f0cb2-8d4a-47ab-ab79-fb1a54109e48" (UID: "688f0cb2-8d4a-47ab-ab79-fb1a54109e48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:14.828363 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.828310    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 20:12:14.888343 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.888312    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-token\" (UniqueName: \"kubernetes.io/projected/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-token\") pod \"30b8551c-d7a1-4527-bd33-6f16c63ad713\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") "
Oct 30 20:12:14.888529 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.888381    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-credential-socket\") pod \"30b8551c-d7a1-4527-bd33-6f16c63ad713\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") "
Oct 30 20:12:14.888529 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.888426    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-data\") pod \"30b8551c-d7a1-4527-bd33-6f16c63ad713\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") "
Oct 30 20:12:14.888529 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.888456    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-workload-socket\") pod \"30b8551c-d7a1-4527-bd33-6f16c63ad713\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") "
Oct 30 20:12:14.888704 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.888567    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-workload-certs\") pod \"30b8551c-d7a1-4527-bd33-6f16c63ad713\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") "
Oct 30 20:12:14.888704 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.888602    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-podinfo\") pod \"30b8551c-d7a1-4527-bd33-6f16c63ad713\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") "
Oct 30 20:12:14.888704 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.888636    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b8551c-d7a1-4527-bd33-6f16c63ad713-kube-api-access-czf2b\" (UniqueName: \"kubernetes.io/projected/30b8551c-d7a1-4527-bd33-6f16c63ad713-kube-api-access-czf2b\") pod \"30b8551c-d7a1-4527-bd33-6f16c63ad713\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") "
Oct 30 20:12:14.888704 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.888694    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b8551c-d7a1-4527-bd33-6f16c63ad713-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b8551c-d7a1-4527-bd33-6f16c63ad713-istiod-ca-cert\") pod \"30b8551c-d7a1-4527-bd33-6f16c63ad713\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") "
Oct 30 20:12:14.888912 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.888736    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-envoy\") pod \"30b8551c-d7a1-4527-bd33-6f16c63ad713\" (UID: \"30b8551c-d7a1-4527-bd33-6f16c63ad713\") "
Oct 30 20:12:14.888912 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.888772    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-credential-socket" pod "30b8551c-d7a1-4527-bd33-6f16c63ad713" (UID: "30b8551c-d7a1-4527-bd33-6f16c63ad713"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:14.888912 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.888787    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-data" pod "30b8551c-d7a1-4527-bd33-6f16c63ad713" (UID: "30b8551c-d7a1-4527-bd33-6f16c63ad713"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:14.888912 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.888879    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-workload-certs" pod "30b8551c-d7a1-4527-bd33-6f16c63ad713" (UID: "30b8551c-d7a1-4527-bd33-6f16c63ad713"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:14.889167 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.889056    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-workload-socket" pod "30b8551c-d7a1-4527-bd33-6f16c63ad713" (UID: "30b8551c-d7a1-4527-bd33-6f16c63ad713"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:14.889523 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.889496    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b8551c-d7a1-4527-bd33-6f16c63ad713-istiod-ca-cert" pod "30b8551c-d7a1-4527-bd33-6f16c63ad713" (UID: "30b8551c-d7a1-4527-bd33-6f16c63ad713"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:14.889632 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.889575    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.889632 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.889600    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.889632 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.889616    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.889745 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.889631    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkjc9\" (UniqueName: \"kubernetes.io/projected/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-kube-api-access-nkjc9\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.889745 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.889647    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.889745 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.889661    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.889745 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.889676    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.889745 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.889691    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.889745 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.889705    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.889745 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.889718    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.889745 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.889731    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.889745 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.889744    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.890076 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.889757    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/688f0cb2-8d4a-47ab-ab79-fb1a54109e48-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.891703 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.891671    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-podinfo" pod "30b8551c-d7a1-4527-bd33-6f16c63ad713" (UID: "30b8551c-d7a1-4527-bd33-6f16c63ad713"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:12:14.891867 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.891772    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b8551c-d7a1-4527-bd33-6f16c63ad713-kube-api-access-czf2b" pod "30b8551c-d7a1-4527-bd33-6f16c63ad713" (UID: "30b8551c-d7a1-4527-bd33-6f16c63ad713"). InnerVolumeSpecName "kube-api-access-czf2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:14.892509 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.891886    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-token" pod "30b8551c-d7a1-4527-bd33-6f16c63ad713" (UID: "30b8551c-d7a1-4527-bd33-6f16c63ad713"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:14.892509 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.892472    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-envoy" pod "30b8551c-d7a1-4527-bd33-6f16c63ad713" (UID: "30b8551c-d7a1-4527-bd33-6f16c63ad713"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:14.990922 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.990889    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.990922 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.990919    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czf2b\" (UniqueName: \"kubernetes.io/projected/30b8551c-d7a1-4527-bd33-6f16c63ad713-kube-api-access-czf2b\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.990922 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.990930    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b8551c-d7a1-4527-bd33-6f16c63ad713-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.991187 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.990939    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.991187 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:14.990948    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b8551c-d7a1-4527-bd33-6f16c63ad713-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:15.301253 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.301220    5408 generic.go:356] "Generic (PLEG): container finished" podID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerID="2185aaaf7d5bb4c253ea7068005c6a68475e80a78eea0c86bcd2121a1248a8f4" exitCode=0
Oct 30 20:12:15.301463 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.301285    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"
Oct 30 20:12:15.301463 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.301324    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" event={"ID":"30b8551c-d7a1-4527-bd33-6f16c63ad713","Type":"ContainerDied","Data":"2185aaaf7d5bb4c253ea7068005c6a68475e80a78eea0c86bcd2121a1248a8f4"}
Oct 30 20:12:15.301463 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.301387    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv" event={"ID":"30b8551c-d7a1-4527-bd33-6f16c63ad713","Type":"ContainerDied","Data":"db387da76aac9aa8135df48f122489c4060f7441b4a4ddc5ffa442158603fa9f"}
Oct 30 20:12:15.301463 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.301416    5408 scope.go:122] "RemoveContainer" containerID="2185aaaf7d5bb4c253ea7068005c6a68475e80a78eea0c86bcd2121a1248a8f4"
Oct 30 20:12:15.303790 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.303568    5408 generic.go:356] "Generic (PLEG): container finished" podID="688f0cb2-8d4a-47ab-ab79-fb1a54109e48" containerID="7d7805b2830bcc21f460e41c7be2ef41ba2d2357171e84e806496ca083d7a23c" exitCode=0
Oct 30 20:12:15.303790 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.303720    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq" event={"ID":"688f0cb2-8d4a-47ab-ab79-fb1a54109e48","Type":"ContainerDied","Data":"7d7805b2830bcc21f460e41c7be2ef41ba2d2357171e84e806496ca083d7a23c"}
Oct 30 20:12:15.303790 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.303757    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq" event={"ID":"688f0cb2-8d4a-47ab-ab79-fb1a54109e48","Type":"ContainerDied","Data":"8066dc67c56eac11e272c0d50fde3008ae2f9a7da9ba79c25580890cc07da618"}
Oct 30 20:12:15.304113 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.303807    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"
Oct 30 20:12:15.334679 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.330897    5408 scope.go:122] "RemoveContainer" containerID="2185aaaf7d5bb4c253ea7068005c6a68475e80a78eea0c86bcd2121a1248a8f4"
Oct 30 20:12:15.334679 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:15.332008    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2185aaaf7d5bb4c253ea7068005c6a68475e80a78eea0c86bcd2121a1248a8f4\": container with ID starting with 2185aaaf7d5bb4c253ea7068005c6a68475e80a78eea0c86bcd2121a1248a8f4 not found: ID does not exist" containerID="2185aaaf7d5bb4c253ea7068005c6a68475e80a78eea0c86bcd2121a1248a8f4"
Oct 30 20:12:15.334679 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.332063    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2185aaaf7d5bb4c253ea7068005c6a68475e80a78eea0c86bcd2121a1248a8f4"} err="failed to get container status \"2185aaaf7d5bb4c253ea7068005c6a68475e80a78eea0c86bcd2121a1248a8f4\": rpc error: code = NotFound desc = could not find container \"2185aaaf7d5bb4c253ea7068005c6a68475e80a78eea0c86bcd2121a1248a8f4\": container with ID starting with 2185aaaf7d5bb4c253ea7068005c6a68475e80a78eea0c86bcd2121a1248a8f4 not found: ID does not exist"
Oct 30 20:12:15.334679 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.332089    5408 scope.go:122] "RemoveContainer" containerID="7d7805b2830bcc21f460e41c7be2ef41ba2d2357171e84e806496ca083d7a23c"
Oct 30 20:12:15.393382 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.393353    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"]
Oct 30 20:12:15.413356 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.413324    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-t8lfv"]
Oct 30 20:12:15.420201 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.420119    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"]
Oct 30 20:12:15.422652 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.422608    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-vq9cq"]
Oct 30 20:12:15.428503 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.428389    5408 scope.go:122] "RemoveContainer" containerID="7d7805b2830bcc21f460e41c7be2ef41ba2d2357171e84e806496ca083d7a23c"
Oct 30 20:12:15.428774 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:15.428753    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d7805b2830bcc21f460e41c7be2ef41ba2d2357171e84e806496ca083d7a23c\": container with ID starting with 7d7805b2830bcc21f460e41c7be2ef41ba2d2357171e84e806496ca083d7a23c not found: ID does not exist" containerID="7d7805b2830bcc21f460e41c7be2ef41ba2d2357171e84e806496ca083d7a23c"
Oct 30 20:12:15.428831 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.428782    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d7805b2830bcc21f460e41c7be2ef41ba2d2357171e84e806496ca083d7a23c"} err="failed to get container status \"7d7805b2830bcc21f460e41c7be2ef41ba2d2357171e84e806496ca083d7a23c\": rpc error: code = NotFound desc = could not find container \"7d7805b2830bcc21f460e41c7be2ef41ba2d2357171e84e806496ca083d7a23c\": container with ID starting with 7d7805b2830bcc21f460e41c7be2ef41ba2d2357171e84e806496ca083d7a23c not found: ID does not exist"
Oct 30 20:12:15.534759 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.534729    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" path="/var/lib/kubelet/pods/30b8551c-d7a1-4527-bd33-6f16c63ad713/volumes"
Oct 30 20:12:15.535306 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:15.535291    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="688f0cb2-8d4a-47ab-ab79-fb1a54109e48" path="/var/lib/kubelet/pods/688f0cb2-8d4a-47ab-ab79-fb1a54109e48/volumes"
Oct 30 20:12:16.598631 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:16.598598    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"]
Oct 30 20:12:16.599262 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:16.599230    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v" podUID="50041f27-99a2-47ba-944a-9693459b8c80" containerName="istio-proxy" containerID="cri-o://b60401a9e062499a42d892fe9cda49b560eefad2aabd529d8266bdc21cb1d880" gracePeriod=30
Oct 30 20:12:16.905290 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:16.905197    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z"]
Oct 30 20:12:21.909073 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:21.906324    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z" podUID="7b091309-1316-445d-bfee-4ba4ba8b8319" containerName="payload-pre-processing" containerID="cri-o://c2dccb43c476c622ef1bd711eb1053b9504c7e1335bcbbd4aa6e4610e56990ec" gracePeriod=25
Oct 30 20:12:21.942449 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:21.942417    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"]
Oct 30 20:12:21.942701 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:21.942650    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn" podUID="e8795c70-cfa2-46d3-85a9-3af34e81144d" containerName="istio-proxy" containerID="cri-o://2321955da4c770774302758225887963c857aa8900892b8a2cc4d62e86ff00a5" gracePeriod=30
Oct 30 20:12:22.001828 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.001797    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.098093    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-workload-socket\") pod \"50041f27-99a2-47ba-944a-9693459b8c80\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") "
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.098172    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-workload-certs\") pod \"50041f27-99a2-47ba-944a-9693459b8c80\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") "
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.098232    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50041f27-99a2-47ba-944a-9693459b8c80-kube-api-access-cw6fd\" (UniqueName: \"kubernetes.io/projected/50041f27-99a2-47ba-944a-9693459b8c80-kube-api-access-cw6fd\") pod \"50041f27-99a2-47ba-944a-9693459b8c80\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") "
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.098265    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-istio-envoy\") pod \"50041f27-99a2-47ba-944a-9693459b8c80\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") "
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.098343    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50041f27-99a2-47ba-944a-9693459b8c80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50041f27-99a2-47ba-944a-9693459b8c80-istiod-ca-cert\") pod \"50041f27-99a2-47ba-944a-9693459b8c80\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") "
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.098421    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50041f27-99a2-47ba-944a-9693459b8c80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50041f27-99a2-47ba-944a-9693459b8c80-istio-podinfo\") pod \"50041f27-99a2-47ba-944a-9693459b8c80\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") "
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.098459    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-istio-data\") pod \"50041f27-99a2-47ba-944a-9693459b8c80\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") "
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.098500    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50041f27-99a2-47ba-944a-9693459b8c80-istio-token\" (UniqueName: \"kubernetes.io/projected/50041f27-99a2-47ba-944a-9693459b8c80-istio-token\") pod \"50041f27-99a2-47ba-944a-9693459b8c80\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") "
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.098653    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-credential-socket\") pod \"50041f27-99a2-47ba-944a-9693459b8c80\" (UID: \"50041f27-99a2-47ba-944a-9693459b8c80\") "
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.099949    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-workload-certs" pod "50041f27-99a2-47ba-944a-9693459b8c80" (UID: "50041f27-99a2-47ba-944a-9693459b8c80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.100203    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-workload-socket" pod "50041f27-99a2-47ba-944a-9693459b8c80" (UID: "50041f27-99a2-47ba-944a-9693459b8c80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.100487    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50041f27-99a2-47ba-944a-9693459b8c80-istiod-ca-cert" pod "50041f27-99a2-47ba-944a-9693459b8c80" (UID: "50041f27-99a2-47ba-944a-9693459b8c80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.100718    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-istio-data" pod "50041f27-99a2-47ba-944a-9693459b8c80" (UID: "50041f27-99a2-47ba-944a-9693459b8c80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.101374    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-credential-socket" pod "50041f27-99a2-47ba-944a-9693459b8c80" (UID: "50041f27-99a2-47ba-944a-9693459b8c80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.101485    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.101504    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.101519    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.101531    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50041f27-99a2-47ba-944a-9693459b8c80-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.101546    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:22.107599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.107533    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50041f27-99a2-47ba-944a-9693459b8c80-istio-token" pod "50041f27-99a2-47ba-944a-9693459b8c80" (UID: "50041f27-99a2-47ba-944a-9693459b8c80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:22.109431 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.109391    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50041f27-99a2-47ba-944a-9693459b8c80-istio-podinfo" pod "50041f27-99a2-47ba-944a-9693459b8c80" (UID: "50041f27-99a2-47ba-944a-9693459b8c80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:12:22.112198 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.111117    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50041f27-99a2-47ba-944a-9693459b8c80-kube-api-access-cw6fd" pod "50041f27-99a2-47ba-944a-9693459b8c80" (UID: "50041f27-99a2-47ba-944a-9693459b8c80"). InnerVolumeSpecName "kube-api-access-cw6fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:22.112198 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.111150    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-istio-envoy" pod "50041f27-99a2-47ba-944a-9693459b8c80" (UID: "50041f27-99a2-47ba-944a-9693459b8c80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:22.289055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.286057    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw6fd\" (UniqueName: \"kubernetes.io/projected/50041f27-99a2-47ba-944a-9693459b8c80-kube-api-access-cw6fd\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:22.289055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.286094    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50041f27-99a2-47ba-944a-9693459b8c80-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:22.289055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.286107    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50041f27-99a2-47ba-944a-9693459b8c80-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:22.289055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.286119    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50041f27-99a2-47ba-944a-9693459b8c80-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:22.363080 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.362646    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z"
Oct 30 20:12:22.488752 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.488713    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b091309-1316-445d-bfee-4ba4ba8b8319-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b091309-1316-445d-bfee-4ba4ba8b8319-plugins-config-volume\") pod \"7b091309-1316-445d-bfee-4ba4ba8b8319\" (UID: \"7b091309-1316-445d-bfee-4ba4ba8b8319\") "
Oct 30 20:12:22.488920 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.488865    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b091309-1316-445d-bfee-4ba4ba8b8319-kube-api-access-qdzj5\" (UniqueName: \"kubernetes.io/projected/7b091309-1316-445d-bfee-4ba4ba8b8319-kube-api-access-qdzj5\") pod \"7b091309-1316-445d-bfee-4ba4ba8b8319\" (UID: \"7b091309-1316-445d-bfee-4ba4ba8b8319\") "
Oct 30 20:12:22.489242 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.489211    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b091309-1316-445d-bfee-4ba4ba8b8319-plugins-config-volume" pod "7b091309-1316-445d-bfee-4ba4ba8b8319" (UID: "7b091309-1316-445d-bfee-4ba4ba8b8319"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:22.489620 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.489597    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b091309-1316-445d-bfee-4ba4ba8b8319-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:22.491349 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.491319    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b091309-1316-445d-bfee-4ba4ba8b8319-kube-api-access-qdzj5" pod "7b091309-1316-445d-bfee-4ba4ba8b8319" (UID: "7b091309-1316-445d-bfee-4ba4ba8b8319"). InnerVolumeSpecName "kube-api-access-qdzj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:22.507395 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.507362    5408 generic.go:356] "Generic (PLEG): container finished" podID="7b091309-1316-445d-bfee-4ba4ba8b8319" containerID="c2dccb43c476c622ef1bd711eb1053b9504c7e1335bcbbd4aa6e4610e56990ec" exitCode=0
Oct 30 20:12:22.507540 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.507488    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z" event={"ID":"7b091309-1316-445d-bfee-4ba4ba8b8319","Type":"ContainerDied","Data":"c2dccb43c476c622ef1bd711eb1053b9504c7e1335bcbbd4aa6e4610e56990ec"}
Oct 30 20:12:22.507540 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.507538    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z" event={"ID":"7b091309-1316-445d-bfee-4ba4ba8b8319","Type":"ContainerDied","Data":"bccd8f6848d39c8bc25b3846adb4deda023401192ab274d8a6c0e19e21ec21d5"}
Oct 30 20:12:22.507667 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.507565    5408 scope.go:122] "RemoveContainer" containerID="c2dccb43c476c622ef1bd711eb1053b9504c7e1335bcbbd4aa6e4610e56990ec"
Oct 30 20:12:22.507726 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.507693    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z"
Oct 30 20:12:22.512591 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.512564    5408 generic.go:356] "Generic (PLEG): container finished" podID="50041f27-99a2-47ba-944a-9693459b8c80" containerID="b60401a9e062499a42d892fe9cda49b560eefad2aabd529d8266bdc21cb1d880" exitCode=0
Oct 30 20:12:22.512734 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.512716    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v" event={"ID":"50041f27-99a2-47ba-944a-9693459b8c80","Type":"ContainerDied","Data":"b60401a9e062499a42d892fe9cda49b560eefad2aabd529d8266bdc21cb1d880"}
Oct 30 20:12:22.512817 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.512765    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v" event={"ID":"50041f27-99a2-47ba-944a-9693459b8c80","Type":"ContainerDied","Data":"b6c2bc96ec995ca4ca845caeb5762eb5fb2d943b7ba8e8f855551cf6297c1a6e"}
Oct 30 20:12:22.512881 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.512842    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"
Oct 30 20:12:22.532174 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.527577    5408 scope.go:122] "RemoveContainer" containerID="c2dccb43c476c622ef1bd711eb1053b9504c7e1335bcbbd4aa6e4610e56990ec"
Oct 30 20:12:22.532174 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:22.527937    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2dccb43c476c622ef1bd711eb1053b9504c7e1335bcbbd4aa6e4610e56990ec\": container with ID starting with c2dccb43c476c622ef1bd711eb1053b9504c7e1335bcbbd4aa6e4610e56990ec not found: ID does not exist" containerID="c2dccb43c476c622ef1bd711eb1053b9504c7e1335bcbbd4aa6e4610e56990ec"
Oct 30 20:12:22.532174 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.527969    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2dccb43c476c622ef1bd711eb1053b9504c7e1335bcbbd4aa6e4610e56990ec"} err="failed to get container status \"c2dccb43c476c622ef1bd711eb1053b9504c7e1335bcbbd4aa6e4610e56990ec\": rpc error: code = NotFound desc = could not find container \"c2dccb43c476c622ef1bd711eb1053b9504c7e1335bcbbd4aa6e4610e56990ec\": container with ID starting with c2dccb43c476c622ef1bd711eb1053b9504c7e1335bcbbd4aa6e4610e56990ec not found: ID does not exist"
Oct 30 20:12:22.532174 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.527992    5408 scope.go:122] "RemoveContainer" containerID="b60401a9e062499a42d892fe9cda49b560eefad2aabd529d8266bdc21cb1d880"
Oct 30 20:12:22.543231 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.543180    5408 scope.go:122] "RemoveContainer" containerID="538e6c2c3732c0219e5b2d83dae3cf81bb5381664cece705c695da5c7ae4970b"
Oct 30 20:12:22.555744 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.555724    5408 scope.go:122] "RemoveContainer" containerID="b60401a9e062499a42d892fe9cda49b560eefad2aabd529d8266bdc21cb1d880"
Oct 30 20:12:22.556124 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:22.556074    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60401a9e062499a42d892fe9cda49b560eefad2aabd529d8266bdc21cb1d880\": container with ID starting with b60401a9e062499a42d892fe9cda49b560eefad2aabd529d8266bdc21cb1d880 not found: ID does not exist" containerID="b60401a9e062499a42d892fe9cda49b560eefad2aabd529d8266bdc21cb1d880"
Oct 30 20:12:22.556124 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.556100    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60401a9e062499a42d892fe9cda49b560eefad2aabd529d8266bdc21cb1d880"} err="failed to get container status \"b60401a9e062499a42d892fe9cda49b560eefad2aabd529d8266bdc21cb1d880\": rpc error: code = NotFound desc = could not find container \"b60401a9e062499a42d892fe9cda49b560eefad2aabd529d8266bdc21cb1d880\": container with ID starting with b60401a9e062499a42d892fe9cda49b560eefad2aabd529d8266bdc21cb1d880 not found: ID does not exist"
Oct 30 20:12:22.556124 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.556120    5408 scope.go:122] "RemoveContainer" containerID="538e6c2c3732c0219e5b2d83dae3cf81bb5381664cece705c695da5c7ae4970b"
Oct 30 20:12:22.556407 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:22.556386    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538e6c2c3732c0219e5b2d83dae3cf81bb5381664cece705c695da5c7ae4970b\": container with ID starting with 538e6c2c3732c0219e5b2d83dae3cf81bb5381664cece705c695da5c7ae4970b not found: ID does not exist" containerID="538e6c2c3732c0219e5b2d83dae3cf81bb5381664cece705c695da5c7ae4970b"
Oct 30 20:12:22.556492 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.556411    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538e6c2c3732c0219e5b2d83dae3cf81bb5381664cece705c695da5c7ae4970b"} err="failed to get container status \"538e6c2c3732c0219e5b2d83dae3cf81bb5381664cece705c695da5c7ae4970b\": rpc error: code = NotFound desc = could not find container \"538e6c2c3732c0219e5b2d83dae3cf81bb5381664cece705c695da5c7ae4970b\": container with ID starting with 538e6c2c3732c0219e5b2d83dae3cf81bb5381664cece705c695da5c7ae4970b not found: ID does not exist"
Oct 30 20:12:22.557207 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.557115    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z"]
Oct 30 20:12:22.564168 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.564144    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-77b4df-c5fcf47ff-6n87z"]
Oct 30 20:12:22.586885 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.586857    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"]
Oct 30 20:12:22.596400 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.591236    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdzj5\" (UniqueName: \"kubernetes.io/projected/7b091309-1316-445d-bfee-4ba4ba8b8319-kube-api-access-qdzj5\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:22.596663 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:22.596639    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-5ls6v"]
Oct 30 20:12:23.532777 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:23.532745    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50041f27-99a2-47ba-944a-9693459b8c80" path="/var/lib/kubelet/pods/50041f27-99a2-47ba-944a-9693459b8c80/volumes"
Oct 30 20:12:23.533277 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:23.533256    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b091309-1316-445d-bfee-4ba4ba8b8319" path="/var/lib/kubelet/pods/7b091309-1316-445d-bfee-4ba4ba8b8319/volumes"
Oct 30 20:12:25.238700 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:25.238663    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll"]
Oct 30 20:12:25.272703 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:25.272659    5408 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:12:25.272893 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:25.272737    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn" podUID="e8795c70-cfa2-46d3-85a9-3af34e81144d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:12:25.276939 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:25.276904    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v"]
Oct 30 20:12:27.225885 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.225859    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 20:12:27.299197 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.299105    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9"]
Oct 30 20:12:27.327366 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.327332    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-workload-socket\") pod \"e8795c70-cfa2-46d3-85a9-3af34e81144d\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") "
Oct 30 20:12:27.327548 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.327393    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-envoy\") pod \"e8795c70-cfa2-46d3-85a9-3af34e81144d\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") "
Oct 30 20:12:27.327548 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.327458    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-token\" (UniqueName: \"kubernetes.io/projected/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-token\") pod \"e8795c70-cfa2-46d3-85a9-3af34e81144d\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") "
Oct 30 20:12:27.327697 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.327574    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-data\") pod \"e8795c70-cfa2-46d3-85a9-3af34e81144d\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") "
Oct 30 20:12:27.327697 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.327611    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-workload-certs\") pod \"e8795c70-cfa2-46d3-85a9-3af34e81144d\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") "
Oct 30 20:12:27.327697 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.327633    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8795c70-cfa2-46d3-85a9-3af34e81144d-kube-api-access-mrrbx\" (UniqueName: \"kubernetes.io/projected/e8795c70-cfa2-46d3-85a9-3af34e81144d-kube-api-access-mrrbx\") pod \"e8795c70-cfa2-46d3-85a9-3af34e81144d\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") "
Oct 30 20:12:27.327862 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.327699    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-credential-socket\") pod \"e8795c70-cfa2-46d3-85a9-3af34e81144d\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") "
Oct 30 20:12:27.327862 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.327721    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8795c70-cfa2-46d3-85a9-3af34e81144d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8795c70-cfa2-46d3-85a9-3af34e81144d-istiod-ca-cert\") pod \"e8795c70-cfa2-46d3-85a9-3af34e81144d\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") "
Oct 30 20:12:27.327862 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.327767    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-podinfo\") pod \"e8795c70-cfa2-46d3-85a9-3af34e81144d\" (UID: \"e8795c70-cfa2-46d3-85a9-3af34e81144d\") "
Oct 30 20:12:27.328812 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.328774    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-data" pod "e8795c70-cfa2-46d3-85a9-3af34e81144d" (UID: "e8795c70-cfa2-46d3-85a9-3af34e81144d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:27.328999 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.328979    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-workload-socket" pod "e8795c70-cfa2-46d3-85a9-3af34e81144d" (UID: "e8795c70-cfa2-46d3-85a9-3af34e81144d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:27.331828 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.331595    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-workload-certs" pod "e8795c70-cfa2-46d3-85a9-3af34e81144d" (UID: "e8795c70-cfa2-46d3-85a9-3af34e81144d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:27.331828 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.331754    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8795c70-cfa2-46d3-85a9-3af34e81144d-istiod-ca-cert" pod "e8795c70-cfa2-46d3-85a9-3af34e81144d" (UID: "e8795c70-cfa2-46d3-85a9-3af34e81144d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:27.331828 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.331807    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-credential-socket" pod "e8795c70-cfa2-46d3-85a9-3af34e81144d" (UID: "e8795c70-cfa2-46d3-85a9-3af34e81144d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:27.332445 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.332424    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-podinfo" pod "e8795c70-cfa2-46d3-85a9-3af34e81144d" (UID: "e8795c70-cfa2-46d3-85a9-3af34e81144d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:12:27.332840 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.332815    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-envoy" pod "e8795c70-cfa2-46d3-85a9-3af34e81144d" (UID: "e8795c70-cfa2-46d3-85a9-3af34e81144d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:27.336094 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.336061    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-token" pod "e8795c70-cfa2-46d3-85a9-3af34e81144d" (UID: "e8795c70-cfa2-46d3-85a9-3af34e81144d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:27.342071 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.339814    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8795c70-cfa2-46d3-85a9-3af34e81144d-kube-api-access-mrrbx" pod "e8795c70-cfa2-46d3-85a9-3af34e81144d" (UID: "e8795c70-cfa2-46d3-85a9-3af34e81144d"). InnerVolumeSpecName "kube-api-access-mrrbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:27.430147 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.430117    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.430147 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.430144    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.430399 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.430160    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrrbx\" (UniqueName: \"kubernetes.io/projected/e8795c70-cfa2-46d3-85a9-3af34e81144d-kube-api-access-mrrbx\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.430399 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.430171    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.430399 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.430180    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8795c70-cfa2-46d3-85a9-3af34e81144d-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.430399 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.430187    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.430399 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.430196    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.430399 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.430205    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.430399 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.430213    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8795c70-cfa2-46d3-85a9-3af34e81144d-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.683662 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.683573    5408 generic.go:356] "Generic (PLEG): container finished" podID="e8795c70-cfa2-46d3-85a9-3af34e81144d" containerID="2321955da4c770774302758225887963c857aa8900892b8a2cc4d62e86ff00a5" exitCode=0
Oct 30 20:12:27.684128 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.684097    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn" event={"ID":"e8795c70-cfa2-46d3-85a9-3af34e81144d","Type":"ContainerDied","Data":"2321955da4c770774302758225887963c857aa8900892b8a2cc4d62e86ff00a5"}
Oct 30 20:12:27.684357 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.684340    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn" event={"ID":"e8795c70-cfa2-46d3-85a9-3af34e81144d","Type":"ContainerDied","Data":"7bdc339e1f71928f79fc46b047e426e2f24301be8175fde8010b1089b06e40e7"}
Oct 30 20:12:27.684478 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.684465    5408 scope.go:122] "RemoveContainer" containerID="2321955da4c770774302758225887963c857aa8900892b8a2cc4d62e86ff00a5"
Oct 30 20:12:27.689102 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.684719    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"
Oct 30 20:12:27.720756 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.720722    5408 scope.go:122] "RemoveContainer" containerID="2321955da4c770774302758225887963c857aa8900892b8a2cc4d62e86ff00a5"
Oct 30 20:12:27.721515 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:27.721458    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2321955da4c770774302758225887963c857aa8900892b8a2cc4d62e86ff00a5\": container with ID starting with 2321955da4c770774302758225887963c857aa8900892b8a2cc4d62e86ff00a5 not found: ID does not exist" containerID="2321955da4c770774302758225887963c857aa8900892b8a2cc4d62e86ff00a5"
Oct 30 20:12:27.721515 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.721483    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2321955da4c770774302758225887963c857aa8900892b8a2cc4d62e86ff00a5"} err="failed to get container status \"2321955da4c770774302758225887963c857aa8900892b8a2cc4d62e86ff00a5\": rpc error: code = NotFound desc = could not find container \"2321955da4c770774302758225887963c857aa8900892b8a2cc4d62e86ff00a5\": container with ID starting with 2321955da4c770774302758225887963c857aa8900892b8a2cc4d62e86ff00a5 not found: ID does not exist"
Oct 30 20:12:27.746818 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.746788    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"]
Oct 30 20:12:27.753198 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:27.752944    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gtbhn"]
Oct 30 20:12:28.293283 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:28.293256    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"]
Oct 30 20:12:28.294046 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:28.293999    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr" podUID="d7a32800-c4b4-46eb-abcd-fee2c0d88a29" containerName="maas-api" containerID="cri-o://cfed23b36a0f925da80129baa357ef29476423016334000737a89f7a4b9e413d" gracePeriod=30
Oct 30 20:12:28.763434 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:28.759805    5408 generic.go:356] "Generic (PLEG): container finished" podID="d7a32800-c4b4-46eb-abcd-fee2c0d88a29" containerID="cfed23b36a0f925da80129baa357ef29476423016334000737a89f7a4b9e413d" exitCode=0
Oct 30 20:12:28.763434 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:28.760013    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr" event={"ID":"d7a32800-c4b4-46eb-abcd-fee2c0d88a29","Type":"ContainerDied","Data":"cfed23b36a0f925da80129baa357ef29476423016334000737a89f7a4b9e413d"}
Oct 30 20:12:28.820758 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:28.820733    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"
Oct 30 20:12:28.886452 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:28.886416    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-maas-api-tls\") pod \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\" (UID: \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\") "
Oct 30 20:12:28.886615 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:28.886474    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-openshift-service-ca\") pod \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\" (UID: \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\") "
Oct 30 20:12:28.886675 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:28.886659    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-metrics-certs\") pod \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\" (UID: \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\") "
Oct 30 20:12:28.886748 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:28.886736    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-kube-api-access-8kjts\" (UniqueName: \"kubernetes.io/projected/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-kube-api-access-8kjts\") pod \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\" (UID: \"d7a32800-c4b4-46eb-abcd-fee2c0d88a29\") "
Oct 30 20:12:28.887726 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:28.887695    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-openshift-service-ca" pod "d7a32800-c4b4-46eb-abcd-fee2c0d88a29" (UID: "d7a32800-c4b4-46eb-abcd-fee2c0d88a29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:28.889555 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:28.889533    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-metrics-certs" pod "d7a32800-c4b4-46eb-abcd-fee2c0d88a29" (UID: "d7a32800-c4b4-46eb-abcd-fee2c0d88a29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:28.889897 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:28.889868    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-kube-api-access-8kjts" pod "d7a32800-c4b4-46eb-abcd-fee2c0d88a29" (UID: "d7a32800-c4b4-46eb-abcd-fee2c0d88a29"). InnerVolumeSpecName "kube-api-access-8kjts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:28.890448 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:28.890423    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-maas-api-tls" pod "d7a32800-c4b4-46eb-abcd-fee2c0d88a29" (UID: "d7a32800-c4b4-46eb-abcd-fee2c0d88a29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:29.003092 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:29.001082    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:29.003092 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:29.001124    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kjts\" (UniqueName: \"kubernetes.io/projected/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-kube-api-access-8kjts\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:29.003092 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:29.001142    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:29.003092 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:29.001157    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a32800-c4b4-46eb-abcd-fee2c0d88a29-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:29.531720 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:29.531678    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8795c70-cfa2-46d3-85a9-3af34e81144d" path="/var/lib/kubelet/pods/e8795c70-cfa2-46d3-85a9-3af34e81144d/volumes"
Oct 30 20:12:29.771399 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:29.771368    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"
Oct 30 20:12:29.771399 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:29.771376    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr" event={"ID":"d7a32800-c4b4-46eb-abcd-fee2c0d88a29","Type":"ContainerDied","Data":"d12a2f3fd4b503942a00873c7f28609561f76d6b0687efb62462a21429e7a102"}
Oct 30 20:12:29.771699 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:29.771456    5408 scope.go:122] "RemoveContainer" containerID="cfed23b36a0f925da80129baa357ef29476423016334000737a89f7a4b9e413d"
Oct 30 20:12:29.803249 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:29.803214    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"]
Oct 30 20:12:29.807012 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:29.806982    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-f5d176-5865d4f49b-wmwgr"]
Oct 30 20:12:30.242677 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.242634    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll" podUID="a848a44a-9e0a-47aa-811f-5a9930b9bd9f" containerName="payload-processing" containerID="cri-o://98577692088c491a4e4b123552a7ba3b4c21731718062364c3caece3865a6b11" gracePeriod=25
Oct 30 20:12:30.278267 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.278211    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v" podUID="c0bb64d8-6abf-45cc-8feb-d7ba98073055" containerName="payload-pre-processing" containerID="cri-o://266adf1eda38260190b3dc1916ed5c639fbc6085d50fd1ddb19e5f8911187c71" gracePeriod=25
Oct 30 20:12:30.568672 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.568646    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll"
Oct 30 20:12:30.627141 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.627109    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v"
Oct 30 20:12:30.627645 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.627606    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a848a44a-9e0a-47aa-811f-5a9930b9bd9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a848a44a-9e0a-47aa-811f-5a9930b9bd9f-plugins-config-volume\") pod \"a848a44a-9e0a-47aa-811f-5a9930b9bd9f\" (UID: \"a848a44a-9e0a-47aa-811f-5a9930b9bd9f\") "
Oct 30 20:12:30.627797 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.627701    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a848a44a-9e0a-47aa-811f-5a9930b9bd9f-kube-api-access-dvn9t\" (UniqueName: \"kubernetes.io/projected/a848a44a-9e0a-47aa-811f-5a9930b9bd9f-kube-api-access-dvn9t\") pod \"a848a44a-9e0a-47aa-811f-5a9930b9bd9f\" (UID: \"a848a44a-9e0a-47aa-811f-5a9930b9bd9f\") "
Oct 30 20:12:30.628359 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.628324    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a848a44a-9e0a-47aa-811f-5a9930b9bd9f-plugins-config-volume" pod "a848a44a-9e0a-47aa-811f-5a9930b9bd9f" (UID: "a848a44a-9e0a-47aa-811f-5a9930b9bd9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:30.630494 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.630466    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a848a44a-9e0a-47aa-811f-5a9930b9bd9f-kube-api-access-dvn9t" pod "a848a44a-9e0a-47aa-811f-5a9930b9bd9f" (UID: "a848a44a-9e0a-47aa-811f-5a9930b9bd9f"). InnerVolumeSpecName "kube-api-access-dvn9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:30.729146 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.729117    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0bb64d8-6abf-45cc-8feb-d7ba98073055-kube-api-access-5fsz4\" (UniqueName: \"kubernetes.io/projected/c0bb64d8-6abf-45cc-8feb-d7ba98073055-kube-api-access-5fsz4\") pod \"c0bb64d8-6abf-45cc-8feb-d7ba98073055\" (UID: \"c0bb64d8-6abf-45cc-8feb-d7ba98073055\") "
Oct 30 20:12:30.729321 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.729242    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0bb64d8-6abf-45cc-8feb-d7ba98073055-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0bb64d8-6abf-45cc-8feb-d7ba98073055-plugins-config-volume\") pod \"c0bb64d8-6abf-45cc-8feb-d7ba98073055\" (UID: \"c0bb64d8-6abf-45cc-8feb-d7ba98073055\") "
Oct 30 20:12:30.729632 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.729606    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bb64d8-6abf-45cc-8feb-d7ba98073055-plugins-config-volume" pod "c0bb64d8-6abf-45cc-8feb-d7ba98073055" (UID: "c0bb64d8-6abf-45cc-8feb-d7ba98073055"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:30.729771 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.729755    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a848a44a-9e0a-47aa-811f-5a9930b9bd9f-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:30.729842 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.729777    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvn9t\" (UniqueName: \"kubernetes.io/projected/a848a44a-9e0a-47aa-811f-5a9930b9bd9f-kube-api-access-dvn9t\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:30.729842 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.729792    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0bb64d8-6abf-45cc-8feb-d7ba98073055-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:30.731209 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.731186    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bb64d8-6abf-45cc-8feb-d7ba98073055-kube-api-access-5fsz4" pod "c0bb64d8-6abf-45cc-8feb-d7ba98073055" (UID: "c0bb64d8-6abf-45cc-8feb-d7ba98073055"). InnerVolumeSpecName "kube-api-access-5fsz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:30.780567 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.780488    5408 generic.go:356] "Generic (PLEG): container finished" podID="c0bb64d8-6abf-45cc-8feb-d7ba98073055" containerID="266adf1eda38260190b3dc1916ed5c639fbc6085d50fd1ddb19e5f8911187c71" exitCode=0
Oct 30 20:12:30.780567 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.780534    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v"
Oct 30 20:12:30.780764 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.780569    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v" event={"ID":"c0bb64d8-6abf-45cc-8feb-d7ba98073055","Type":"ContainerDied","Data":"266adf1eda38260190b3dc1916ed5c639fbc6085d50fd1ddb19e5f8911187c71"}
Oct 30 20:12:30.780764 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.780616    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v" event={"ID":"c0bb64d8-6abf-45cc-8feb-d7ba98073055","Type":"ContainerDied","Data":"4d65213267e6e089d6c14194dde014c2d4f9cd1a4ea37904c528f3a3c79ae446"}
Oct 30 20:12:30.780764 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.780628    5408 scope.go:122] "RemoveContainer" containerID="266adf1eda38260190b3dc1916ed5c639fbc6085d50fd1ddb19e5f8911187c71"
Oct 30 20:12:30.782195 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.782171    5408 generic.go:356] "Generic (PLEG): container finished" podID="a848a44a-9e0a-47aa-811f-5a9930b9bd9f" containerID="98577692088c491a4e4b123552a7ba3b4c21731718062364c3caece3865a6b11" exitCode=0
Oct 30 20:12:30.782281 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.782207    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll"
Oct 30 20:12:30.782281 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.782262    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll" event={"ID":"a848a44a-9e0a-47aa-811f-5a9930b9bd9f","Type":"ContainerDied","Data":"98577692088c491a4e4b123552a7ba3b4c21731718062364c3caece3865a6b11"}
Oct 30 20:12:30.782393 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.782313    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll" event={"ID":"a848a44a-9e0a-47aa-811f-5a9930b9bd9f","Type":"ContainerDied","Data":"7a9d87d1cdb6b555c9801d283545d5b52371fe55024b27c2991bdd36635e3f03"}
Oct 30 20:12:30.792202 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.792185    5408 scope.go:122] "RemoveContainer" containerID="266adf1eda38260190b3dc1916ed5c639fbc6085d50fd1ddb19e5f8911187c71"
Oct 30 20:12:30.792520 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:30.792491    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"266adf1eda38260190b3dc1916ed5c639fbc6085d50fd1ddb19e5f8911187c71\": container with ID starting with 266adf1eda38260190b3dc1916ed5c639fbc6085d50fd1ddb19e5f8911187c71 not found: ID does not exist" containerID="266adf1eda38260190b3dc1916ed5c639fbc6085d50fd1ddb19e5f8911187c71"
Oct 30 20:12:30.792618 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.792531    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"266adf1eda38260190b3dc1916ed5c639fbc6085d50fd1ddb19e5f8911187c71"} err="failed to get container status \"266adf1eda38260190b3dc1916ed5c639fbc6085d50fd1ddb19e5f8911187c71\": rpc error: code = NotFound desc = could not find container \"266adf1eda38260190b3dc1916ed5c639fbc6085d50fd1ddb19e5f8911187c71\": container with ID starting with 266adf1eda38260190b3dc1916ed5c639fbc6085d50fd1ddb19e5f8911187c71 not found: ID does not exist"
Oct 30 20:12:30.792618 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.792556    5408 scope.go:122] "RemoveContainer" containerID="98577692088c491a4e4b123552a7ba3b4c21731718062364c3caece3865a6b11"
Oct 30 20:12:30.801795 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.801778    5408 scope.go:122] "RemoveContainer" containerID="98577692088c491a4e4b123552a7ba3b4c21731718062364c3caece3865a6b11"
Oct 30 20:12:30.802078 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:30.802059    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98577692088c491a4e4b123552a7ba3b4c21731718062364c3caece3865a6b11\": container with ID starting with 98577692088c491a4e4b123552a7ba3b4c21731718062364c3caece3865a6b11 not found: ID does not exist" containerID="98577692088c491a4e4b123552a7ba3b4c21731718062364c3caece3865a6b11"
Oct 30 20:12:30.802139 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.802086    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98577692088c491a4e4b123552a7ba3b4c21731718062364c3caece3865a6b11"} err="failed to get container status \"98577692088c491a4e4b123552a7ba3b4c21731718062364c3caece3865a6b11\": rpc error: code = NotFound desc = could not find container \"98577692088c491a4e4b123552a7ba3b4c21731718062364c3caece3865a6b11\": container with ID starting with 98577692088c491a4e4b123552a7ba3b4c21731718062364c3caece3865a6b11 not found: ID does not exist"
Oct 30 20:12:30.807780 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.807760    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v"]
Oct 30 20:12:30.812646 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.812626    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f5d176-79cb4b6967-m5b5v"]
Oct 30 20:12:30.827603 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.827583    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll"]
Oct 30 20:12:30.831525 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.831503    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fsz4\" (UniqueName: \"kubernetes.io/projected/c0bb64d8-6abf-45cc-8feb-d7ba98073055-kube-api-access-5fsz4\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:30.833478 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:30.833459    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f5d176-85fd754896-cqsll"]
Oct 30 20:12:31.532057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:31.532014    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a848a44a-9e0a-47aa-811f-5a9930b9bd9f" path="/var/lib/kubelet/pods/a848a44a-9e0a-47aa-811f-5a9930b9bd9f/volumes"
Oct 30 20:12:31.532451 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:31.532438    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0bb64d8-6abf-45cc-8feb-d7ba98073055" path="/var/lib/kubelet/pods/c0bb64d8-6abf-45cc-8feb-d7ba98073055/volumes"
Oct 30 20:12:31.532840 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:31.532828    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7a32800-c4b4-46eb-abcd-fee2c0d88a29" path="/var/lib/kubelet/pods/d7a32800-c4b4-46eb-abcd-fee2c0d88a29/volumes"
Oct 30 20:12:32.299610 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:32.299409    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9" podUID="c7bf362b-a122-4417-813a-1c1f30c05297" containerName="payload-pre-processing" containerID="cri-o://044e23d5a4340c49cbe529d7f2241020a33be6ec3068162ce04592a3f837081d" gracePeriod=25
Oct 30 20:12:32.638282 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:32.638250    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9"
Oct 30 20:12:32.790243 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:32.790209    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7bf362b-a122-4417-813a-1c1f30c05297-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7bf362b-a122-4417-813a-1c1f30c05297-plugins-config-volume\") pod \"c7bf362b-a122-4417-813a-1c1f30c05297\" (UID: \"c7bf362b-a122-4417-813a-1c1f30c05297\") "
Oct 30 20:12:32.790415 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:32.790260    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7bf362b-a122-4417-813a-1c1f30c05297-kube-api-access-654jk\" (UniqueName: \"kubernetes.io/projected/c7bf362b-a122-4417-813a-1c1f30c05297-kube-api-access-654jk\") pod \"c7bf362b-a122-4417-813a-1c1f30c05297\" (UID: \"c7bf362b-a122-4417-813a-1c1f30c05297\") "
Oct 30 20:12:32.790597 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:32.790571    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7bf362b-a122-4417-813a-1c1f30c05297-plugins-config-volume" pod "c7bf362b-a122-4417-813a-1c1f30c05297" (UID: "c7bf362b-a122-4417-813a-1c1f30c05297"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:32.790995 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:32.790966    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7bf362b-a122-4417-813a-1c1f30c05297-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:32.792755 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:32.792726    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7bf362b-a122-4417-813a-1c1f30c05297-kube-api-access-654jk" pod "c7bf362b-a122-4417-813a-1c1f30c05297" (UID: "c7bf362b-a122-4417-813a-1c1f30c05297"). InnerVolumeSpecName "kube-api-access-654jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:32.803752 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:32.803728    5408 generic.go:356] "Generic (PLEG): container finished" podID="c7bf362b-a122-4417-813a-1c1f30c05297" containerID="044e23d5a4340c49cbe529d7f2241020a33be6ec3068162ce04592a3f837081d" exitCode=0
Oct 30 20:12:32.803870 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:32.803776    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9"
Oct 30 20:12:32.803870 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:32.803803    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9" event={"ID":"c7bf362b-a122-4417-813a-1c1f30c05297","Type":"ContainerDied","Data":"044e23d5a4340c49cbe529d7f2241020a33be6ec3068162ce04592a3f837081d"}
Oct 30 20:12:32.804054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:32.803869    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9" event={"ID":"c7bf362b-a122-4417-813a-1c1f30c05297","Type":"ContainerDied","Data":"16f33a176cefd41b0b68e5fb1ffcddec8f52727a196325fd564e219e00eee2f7"}
Oct 30 20:12:32.804054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:32.803898    5408 scope.go:122] "RemoveContainer" containerID="044e23d5a4340c49cbe529d7f2241020a33be6ec3068162ce04592a3f837081d"
Oct 30 20:12:32.815023 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:32.815003    5408 scope.go:122] "RemoveContainer" containerID="044e23d5a4340c49cbe529d7f2241020a33be6ec3068162ce04592a3f837081d"
Oct 30 20:12:32.815325 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:32.815302    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"044e23d5a4340c49cbe529d7f2241020a33be6ec3068162ce04592a3f837081d\": container with ID starting with 044e23d5a4340c49cbe529d7f2241020a33be6ec3068162ce04592a3f837081d not found: ID does not exist" containerID="044e23d5a4340c49cbe529d7f2241020a33be6ec3068162ce04592a3f837081d"
Oct 30 20:12:32.815410 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:32.815330    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"044e23d5a4340c49cbe529d7f2241020a33be6ec3068162ce04592a3f837081d"} err="failed to get container status \"044e23d5a4340c49cbe529d7f2241020a33be6ec3068162ce04592a3f837081d\": rpc error: code = NotFound desc = could not find container \"044e23d5a4340c49cbe529d7f2241020a33be6ec3068162ce04592a3f837081d\": container with ID starting with 044e23d5a4340c49cbe529d7f2241020a33be6ec3068162ce04592a3f837081d not found: ID does not exist"
Oct 30 20:12:32.832602 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:32.832576    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9"]
Oct 30 20:12:32.834809 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:32.834787    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b0d13a-c864c8bdc-vtbr9"]
Oct 30 20:12:32.892017 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:32.891950    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-654jk\" (UniqueName: \"kubernetes.io/projected/c7bf362b-a122-4417-813a-1c1f30c05297-kube-api-access-654jk\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:33.533623 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:33.533593    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7bf362b-a122-4417-813a-1c1f30c05297" path="/var/lib/kubelet/pods/c7bf362b-a122-4417-813a-1c1f30c05297/volumes"
Oct 30 20:12:34.117500 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.117461    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"]
Oct 30 20:12:34.117836 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.117726    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9" podUID="eb4fd624-deaf-4da1-9183-e83b29f52d64" containerName="maas-api" containerID="cri-o://1c55605f8efa91f700757e87e914b4d5cb65af67a8a3015a59c44e667b80f886" gracePeriod=30
Oct 30 20:12:34.450117 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.446308    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"]
Oct 30 20:12:34.462051 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.451849    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"]
Oct 30 20:12:34.510057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.495727    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"]
Oct 30 20:12:34.647926 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.647899    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 20:12:34.657340 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.657315    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb4fd624-deaf-4da1-9183-e83b29f52d64-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4fd624-deaf-4da1-9183-e83b29f52d64-metrics-certs\") pod \"eb4fd624-deaf-4da1-9183-e83b29f52d64\" (UID: \"eb4fd624-deaf-4da1-9183-e83b29f52d64\") "
Oct 30 20:12:34.657496 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.657397    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb4fd624-deaf-4da1-9183-e83b29f52d64-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4fd624-deaf-4da1-9183-e83b29f52d64-maas-api-tls\") pod \"eb4fd624-deaf-4da1-9183-e83b29f52d64\" (UID: \"eb4fd624-deaf-4da1-9183-e83b29f52d64\") "
Oct 30 20:12:34.657496 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.657445    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb4fd624-deaf-4da1-9183-e83b29f52d64-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4fd624-deaf-4da1-9183-e83b29f52d64-openshift-service-ca\") pod \"eb4fd624-deaf-4da1-9183-e83b29f52d64\" (UID: \"eb4fd624-deaf-4da1-9183-e83b29f52d64\") "
Oct 30 20:12:34.657496 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.657484    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb4fd624-deaf-4da1-9183-e83b29f52d64-kube-api-access-n649j\" (UniqueName: \"kubernetes.io/projected/eb4fd624-deaf-4da1-9183-e83b29f52d64-kube-api-access-n649j\") pod \"eb4fd624-deaf-4da1-9183-e83b29f52d64\" (UID: \"eb4fd624-deaf-4da1-9183-e83b29f52d64\") "
Oct 30 20:12:34.657971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.657940    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb4fd624-deaf-4da1-9183-e83b29f52d64-openshift-service-ca" pod "eb4fd624-deaf-4da1-9183-e83b29f52d64" (UID: "eb4fd624-deaf-4da1-9183-e83b29f52d64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:34.658148 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.658110    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4fd624-deaf-4da1-9183-e83b29f52d64-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:34.659631 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.659608    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb4fd624-deaf-4da1-9183-e83b29f52d64-metrics-certs" pod "eb4fd624-deaf-4da1-9183-e83b29f52d64" (UID: "eb4fd624-deaf-4da1-9183-e83b29f52d64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:34.659768 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.659750    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb4fd624-deaf-4da1-9183-e83b29f52d64-maas-api-tls" pod "eb4fd624-deaf-4da1-9183-e83b29f52d64" (UID: "eb4fd624-deaf-4da1-9183-e83b29f52d64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:34.659959 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.659941    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4fd624-deaf-4da1-9183-e83b29f52d64-kube-api-access-n649j" pod "eb4fd624-deaf-4da1-9183-e83b29f52d64" (UID: "eb4fd624-deaf-4da1-9183-e83b29f52d64"). InnerVolumeSpecName "kube-api-access-n649j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:34.759163 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.759135    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4fd624-deaf-4da1-9183-e83b29f52d64-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:34.759163 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.759160    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4fd624-deaf-4da1-9183-e83b29f52d64-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:34.759423 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.759172    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n649j\" (UniqueName: \"kubernetes.io/projected/eb4fd624-deaf-4da1-9183-e83b29f52d64-kube-api-access-n649j\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:34.821865 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.821828    5408 generic.go:356] "Generic (PLEG): container finished" podID="eb4fd624-deaf-4da1-9183-e83b29f52d64" containerID="1c55605f8efa91f700757e87e914b4d5cb65af67a8a3015a59c44e667b80f886" exitCode=0
Oct 30 20:12:34.822006 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.821877    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"
Oct 30 20:12:34.822006 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.821915    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9" event={"ID":"eb4fd624-deaf-4da1-9183-e83b29f52d64","Type":"ContainerDied","Data":"1c55605f8efa91f700757e87e914b4d5cb65af67a8a3015a59c44e667b80f886"}
Oct 30 20:12:34.822006 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.821968    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9" event={"ID":"eb4fd624-deaf-4da1-9183-e83b29f52d64","Type":"ContainerDied","Data":"56dc46a4098cedd974998de82326b8e3dd76a52ecec84e3c6150892a707326c3"}
Oct 30 20:12:34.822006 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.821990    5408 scope.go:122] "RemoveContainer" containerID="1c55605f8efa91f700757e87e914b4d5cb65af67a8a3015a59c44e667b80f886"
Oct 30 20:12:34.832379 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.832347    5408 scope.go:122] "RemoveContainer" containerID="1c55605f8efa91f700757e87e914b4d5cb65af67a8a3015a59c44e667b80f886"
Oct 30 20:12:34.832653 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:34.832627    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c55605f8efa91f700757e87e914b4d5cb65af67a8a3015a59c44e667b80f886\": container with ID starting with 1c55605f8efa91f700757e87e914b4d5cb65af67a8a3015a59c44e667b80f886 not found: ID does not exist" containerID="1c55605f8efa91f700757e87e914b4d5cb65af67a8a3015a59c44e667b80f886"
Oct 30 20:12:34.832730 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.832667    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c55605f8efa91f700757e87e914b4d5cb65af67a8a3015a59c44e667b80f886"} err="failed to get container status \"1c55605f8efa91f700757e87e914b4d5cb65af67a8a3015a59c44e667b80f886\": rpc error: code = NotFound desc = could not find container \"1c55605f8efa91f700757e87e914b4d5cb65af67a8a3015a59c44e667b80f886\": container with ID starting with 1c55605f8efa91f700757e87e914b4d5cb65af67a8a3015a59c44e667b80f886 not found: ID does not exist"
Oct 30 20:12:34.907050 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.899584    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"]
Oct 30 20:12:34.908313 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:34.908274    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b0d13a-5d549d7f69-8qfx9"]
Oct 30 20:12:35.531841 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:35.531796    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb4fd624-deaf-4da1-9183-e83b29f52d64" path="/var/lib/kubelet/pods/eb4fd624-deaf-4da1-9183-e83b29f52d64/volumes"
Oct 30 20:12:39.662501 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:39.662469    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:12:43.213282 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:43.213242    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"]
Oct 30 20:12:47.208550 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:47.208511    5408 scope.go:122] "RemoveContainer" containerID="d46cc2f75c49f61cf5c9a62bcb8c58b3f142fb3089388e48d2334789f92db9d0"
Oct 30 20:12:47.218328 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:47.218294    5408 scope.go:122] "RemoveContainer" containerID="b911e63d830ac74f035c3c11fe0ef04eb5995e3bc7a44fa85247fb049e4a565e"
Oct 30 20:12:49.562679 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:49.562625    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" containerID="cri-o://5296966643fd29d112dcc525a53ef5a52c7e84bb6ed167fd2cad3d72808230c8" gracePeriod=45
Oct 30 20:12:49.612578 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:49.612530    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main" containerID="cri-o://0d717eb226e8b5c73e64bc593680dcd58c2638f38ebf3798f0ef13e4d07aab0a" gracePeriod=45
Oct 30 20:12:49.637342 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:49.637303    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main" containerID="cri-o://736ccf33812f9fa040dfe650c49a85a2e917017a25af0537e65da7e1547bd8aa" gracePeriod=45
Oct 30 20:12:49.671252 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:49.671222    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": dial tcp 10.134.0.48:8000: connect: connection refused"
Oct 30 20:12:49.995889 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:49.995856    5408 generic.go:356] "Generic (PLEG): container finished" podID="99c65275-2119-45ac-8393-c29bce507f33" containerID="5296966643fd29d112dcc525a53ef5a52c7e84bb6ed167fd2cad3d72808230c8" exitCode=0
Oct 30 20:12:49.996002 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:49.995923    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" event={"ID":"99c65275-2119-45ac-8393-c29bce507f33","Type":"ContainerDied","Data":"5296966643fd29d112dcc525a53ef5a52c7e84bb6ed167fd2cad3d72808230c8"}
Oct 30 20:12:49.996002 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:49.995974    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58" event={"ID":"99c65275-2119-45ac-8393-c29bce507f33","Type":"ContainerDied","Data":"2c7eb4b28af1c173410ab0a49563e48ff17f66b665f42b73944c20a84f8a63fc"}
Oct 30 20:12:49.996002 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:49.995992    5408 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c7eb4b28af1c173410ab0a49563e48ff17f66b665f42b73944c20a84f8a63fc"
Oct 30 20:12:50.002208 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.002169    5408 generic.go:356] "Generic (PLEG): container finished" podID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerID="0d717eb226e8b5c73e64bc593680dcd58c2638f38ebf3798f0ef13e4d07aab0a" exitCode=0
Oct 30 20:12:50.002355 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.002324    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" event={"ID":"76e5be67-6a55-4e8f-94dc-1a64e11d7f65","Type":"ContainerDied","Data":"0d717eb226e8b5c73e64bc593680dcd58c2638f38ebf3798f0ef13e4d07aab0a"}
Oct 30 20:12:50.002483 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.002360    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9" event={"ID":"76e5be67-6a55-4e8f-94dc-1a64e11d7f65","Type":"ContainerDied","Data":"b928f0361d84576174901b7df7ac64f352b6c8b12c565737627e4c5fced32c75"}
Oct 30 20:12:50.002483 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.002378    5408 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b928f0361d84576174901b7df7ac64f352b6c8b12c565737627e4c5fced32c75"
Oct 30 20:12:50.003976 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.003951    5408 generic.go:356] "Generic (PLEG): container finished" podID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerID="736ccf33812f9fa040dfe650c49a85a2e917017a25af0537e65da7e1547bd8aa" exitCode=0
Oct 30 20:12:50.004132 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.004057    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" event={"ID":"910fde4a-f4d1-4547-a060-2cf6888e5942","Type":"ContainerDied","Data":"736ccf33812f9fa040dfe650c49a85a2e917017a25af0537e65da7e1547bd8aa"}
Oct 30 20:12:50.004132 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.004096    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q" event={"ID":"910fde4a-f4d1-4547-a060-2cf6888e5942","Type":"ContainerDied","Data":"a55083a43be6dee80e680b30512e1cd4cece151db8f99ad87bcf3d95daa081de"}
Oct 30 20:12:50.004132 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.004108    5408 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a55083a43be6dee80e680b30512e1cd4cece151db8f99ad87bcf3d95daa081de"
Oct 30 20:12:50.010450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.010428    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 20:12:50.014531 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.014508    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 20:12:50.018234 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.018218    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 20:12:50.129041 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.128942    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-home\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-home\") pod \"99c65275-2119-45ac-8393-c29bce507f33\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") "
Oct 30 20:12:50.129041 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.128986    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-tls-certs\" (UniqueName: \"kubernetes.io/secret/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-tls-certs\") pod \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") "
Oct 30 20:12:50.129279 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129023    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/910fde4a-f4d1-4547-a060-2cf6888e5942-kube-api-access-ttfkc\" (UniqueName: \"kubernetes.io/projected/910fde4a-f4d1-4547-a060-2cf6888e5942-kube-api-access-ttfkc\") pod \"910fde4a-f4d1-4547-a060-2cf6888e5942\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") "
Oct 30 20:12:50.129279 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129158    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-home\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-home\") pod \"910fde4a-f4d1-4547-a060-2cf6888e5942\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") "
Oct 30 20:12:50.129279 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129206    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-kube-api-access-wbhd8\" (UniqueName: \"kubernetes.io/projected/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-kube-api-access-wbhd8\") pod \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") "
Oct 30 20:12:50.129279 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129239    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99c65275-2119-45ac-8393-c29bce507f33-tls-certs\" (UniqueName: \"kubernetes.io/secret/99c65275-2119-45ac-8393-c29bce507f33-tls-certs\") pod \"99c65275-2119-45ac-8393-c29bce507f33\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") "
Oct 30 20:12:50.129538 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129279    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-dshm\") pod \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") "
Oct 30 20:12:50.129538 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129323    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-model-cache\") pod \"910fde4a-f4d1-4547-a060-2cf6888e5942\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") "
Oct 30 20:12:50.129538 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129350    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-home" pod "99c65275-2119-45ac-8393-c29bce507f33" (UID: "99c65275-2119-45ac-8393-c29bce507f33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:50.129538 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129367    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-model-cache\") pod \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") "
Oct 30 20:12:50.129538 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129418    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/910fde4a-f4d1-4547-a060-2cf6888e5942-tls-certs\" (UniqueName: \"kubernetes.io/secret/910fde4a-f4d1-4547-a060-2cf6888e5942-tls-certs\") pod \"910fde4a-f4d1-4547-a060-2cf6888e5942\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") "
Oct 30 20:12:50.130098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129541    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-tmp-dir\") pod \"910fde4a-f4d1-4547-a060-2cf6888e5942\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") "
Oct 30 20:12:50.130098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129577    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-model-cache" pod "76e5be67-6a55-4e8f-94dc-1a64e11d7f65" (UID: "76e5be67-6a55-4e8f-94dc-1a64e11d7f65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:50.130098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129603    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-tmp-dir\") pod \"99c65275-2119-45ac-8393-c29bce507f33\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") "
Oct 30 20:12:50.130098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129639    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-tmp-dir\") pod \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") "
Oct 30 20:12:50.130098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129688    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-model-cache\") pod \"99c65275-2119-45ac-8393-c29bce507f33\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") "
Oct 30 20:12:50.130098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129727    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-dshm\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-dshm\") pod \"910fde4a-f4d1-4547-a060-2cf6888e5942\" (UID: \"910fde4a-f4d1-4547-a060-2cf6888e5942\") "
Oct 30 20:12:50.130098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129761    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-home\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-home\") pod \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\" (UID: \"76e5be67-6a55-4e8f-94dc-1a64e11d7f65\") "
Oct 30 20:12:50.130098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129793    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-dshm\") pod \"99c65275-2119-45ac-8393-c29bce507f33\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") "
Oct 30 20:12:50.130098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129856    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c65275-2119-45ac-8393-c29bce507f33-kube-api-access-sqf9g\" (UniqueName: \"kubernetes.io/projected/99c65275-2119-45ac-8393-c29bce507f33-kube-api-access-sqf9g\") pod \"99c65275-2119-45ac-8393-c29bce507f33\" (UID: \"99c65275-2119-45ac-8393-c29bce507f33\") "
Oct 30 20:12:50.130528 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.130473    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.130528 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.130493    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.130528 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.129855    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-home" pod "910fde4a-f4d1-4547-a060-2cf6888e5942" (UID: "910fde4a-f4d1-4547-a060-2cf6888e5942"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:50.130528 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.130054    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-model-cache" pod "99c65275-2119-45ac-8393-c29bce507f33" (UID: "99c65275-2119-45ac-8393-c29bce507f33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:50.130528 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.130068    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-model-cache" pod "910fde4a-f4d1-4547-a060-2cf6888e5942" (UID: "910fde4a-f4d1-4547-a060-2cf6888e5942"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:50.130528 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.130366    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-tmp-dir" pod "99c65275-2119-45ac-8393-c29bce507f33" (UID: "99c65275-2119-45ac-8393-c29bce507f33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:50.130852 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.130565    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-tmp-dir" pod "910fde4a-f4d1-4547-a060-2cf6888e5942" (UID: "910fde4a-f4d1-4547-a060-2cf6888e5942"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:50.130852 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.130770    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-tmp-dir" pod "76e5be67-6a55-4e8f-94dc-1a64e11d7f65" (UID: "76e5be67-6a55-4e8f-94dc-1a64e11d7f65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:50.133084 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.133023    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-home" pod "76e5be67-6a55-4e8f-94dc-1a64e11d7f65" (UID: "76e5be67-6a55-4e8f-94dc-1a64e11d7f65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:50.133227 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.133129    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-dshm" pod "76e5be67-6a55-4e8f-94dc-1a64e11d7f65" (UID: "76e5be67-6a55-4e8f-94dc-1a64e11d7f65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:50.133227 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.133209    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/910fde4a-f4d1-4547-a060-2cf6888e5942-kube-api-access-ttfkc" pod "910fde4a-f4d1-4547-a060-2cf6888e5942" (UID: "910fde4a-f4d1-4547-a060-2cf6888e5942"). InnerVolumeSpecName "kube-api-access-ttfkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:50.133378 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.133242    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-kube-api-access-wbhd8" pod "76e5be67-6a55-4e8f-94dc-1a64e11d7f65" (UID: "76e5be67-6a55-4e8f-94dc-1a64e11d7f65"). InnerVolumeSpecName "kube-api-access-wbhd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:50.133717 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.133688    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99c65275-2119-45ac-8393-c29bce507f33-tls-certs" pod "99c65275-2119-45ac-8393-c29bce507f33" (UID: "99c65275-2119-45ac-8393-c29bce507f33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:50.134488 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.134457    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c65275-2119-45ac-8393-c29bce507f33-kube-api-access-sqf9g" pod "99c65275-2119-45ac-8393-c29bce507f33" (UID: "99c65275-2119-45ac-8393-c29bce507f33"). InnerVolumeSpecName "kube-api-access-sqf9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:50.134717 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.134696    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-dshm" pod "910fde4a-f4d1-4547-a060-2cf6888e5942" (UID: "910fde4a-f4d1-4547-a060-2cf6888e5942"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:50.134851 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.134800    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/910fde4a-f4d1-4547-a060-2cf6888e5942-tls-certs" pod "910fde4a-f4d1-4547-a060-2cf6888e5942" (UID: "910fde4a-f4d1-4547-a060-2cf6888e5942"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:50.135432 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.135408    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-dshm" pod "99c65275-2119-45ac-8393-c29bce507f33" (UID: "99c65275-2119-45ac-8393-c29bce507f33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:50.135667 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.135647    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-tls-certs" pod "76e5be67-6a55-4e8f-94dc-1a64e11d7f65" (UID: "76e5be67-6a55-4e8f-94dc-1a64e11d7f65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:50.232138 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.232100    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.232138 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.232129    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbhd8\" (UniqueName: \"kubernetes.io/projected/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-kube-api-access-wbhd8\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.232138 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.232140    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c65275-2119-45ac-8393-c29bce507f33-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.232138 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.232149    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.232511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.232158    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.232511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.232166    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910fde4a-f4d1-4547-a060-2cf6888e5942-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.232511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.232174    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.232511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.232182    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.232511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.232189    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.232511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.232196    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.232511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.232204    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/910fde4a-f4d1-4547-a060-2cf6888e5942-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.232511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.232211    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.232511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.232218    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c65275-2119-45ac-8393-c29bce507f33-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.232511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.232226    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqf9g\" (UniqueName: \"kubernetes.io/projected/99c65275-2119-45ac-8393-c29bce507f33-kube-api-access-sqf9g\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.232511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.232234    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e5be67-6a55-4e8f-94dc-1a64e11d7f65-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:50.232511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:50.232242    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttfkc\" (UniqueName: \"kubernetes.io/projected/910fde4a-f4d1-4547-a060-2cf6888e5942-kube-api-access-ttfkc\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:51.010501 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:51.010468    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"
Oct 30 20:12:51.010996 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:51.010468    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"
Oct 30 20:12:51.010996 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:51.010480    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"
Oct 30 20:12:51.053149 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:51.053114    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"]
Oct 30 20:12:51.057246 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:51.057216    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/distinct-w5-278ffa-kserve-5995c5f648-hfjm9"]
Oct 30 20:12:51.074095 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:51.074064    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"]
Oct 30 20:12:51.080891 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:51.080868    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/simulator-w5-278ffa-kserve-66c45dc6d-59l8q"]
Oct 30 20:12:51.099460 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:51.099438    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"]
Oct 30 20:12:51.106457 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:51.106434    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/premium-w5-278ffa-kserve-f4f68c54f-89s58"]
Oct 30 20:12:51.531197 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:51.531168    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" path="/var/lib/kubelet/pods/76e5be67-6a55-4e8f-94dc-1a64e11d7f65/volumes"
Oct 30 20:12:51.531595 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:51.531582    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" path="/var/lib/kubelet/pods/910fde4a-f4d1-4547-a060-2cf6888e5942/volumes"
Oct 30 20:12:51.531963 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:51.531951    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99c65275-2119-45ac-8393-c29bce507f33" path="/var/lib/kubelet/pods/99c65275-2119-45ac-8393-c29bce507f33/volumes"
Oct 30 20:12:52.741455 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:52.741419    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"]
Oct 30 20:12:52.741870 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:52.741713    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8" podUID="25efa968-ddb4-4893-8209-176519fb0224" containerName="maas-api" containerID="cri-o://402d598b56ba3b38e165b41c30b8e8f9d111fde41e99a3784619b59bd57e9b2f" gracePeriod=30
Oct 30 20:12:53.151401 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:53.151316    5408 generic.go:356] "Generic (PLEG): container finished" podID="25efa968-ddb4-4893-8209-176519fb0224" containerID="402d598b56ba3b38e165b41c30b8e8f9d111fde41e99a3784619b59bd57e9b2f" exitCode=0
Oct 30 20:12:53.151562 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:53.151500    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8" event={"ID":"25efa968-ddb4-4893-8209-176519fb0224","Type":"ContainerDied","Data":"402d598b56ba3b38e165b41c30b8e8f9d111fde41e99a3784619b59bd57e9b2f"}
Oct 30 20:12:53.196640 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:53.196618    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"
Oct 30 20:12:53.242554 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:53.242520    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25efa968-ddb4-4893-8209-176519fb0224-kube-api-access-2vxqw\" (UniqueName: \"kubernetes.io/projected/25efa968-ddb4-4893-8209-176519fb0224-kube-api-access-2vxqw\") pod \"25efa968-ddb4-4893-8209-176519fb0224\" (UID: \"25efa968-ddb4-4893-8209-176519fb0224\") "
Oct 30 20:12:53.242735 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:53.242589    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25efa968-ddb4-4893-8209-176519fb0224-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25efa968-ddb4-4893-8209-176519fb0224-metrics-certs\") pod \"25efa968-ddb4-4893-8209-176519fb0224\" (UID: \"25efa968-ddb4-4893-8209-176519fb0224\") "
Oct 30 20:12:53.242735 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:53.242646    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25efa968-ddb4-4893-8209-176519fb0224-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25efa968-ddb4-4893-8209-176519fb0224-maas-api-tls\") pod \"25efa968-ddb4-4893-8209-176519fb0224\" (UID: \"25efa968-ddb4-4893-8209-176519fb0224\") "
Oct 30 20:12:53.242735 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:53.242707    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25efa968-ddb4-4893-8209-176519fb0224-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25efa968-ddb4-4893-8209-176519fb0224-openshift-service-ca\") pod \"25efa968-ddb4-4893-8209-176519fb0224\" (UID: \"25efa968-ddb4-4893-8209-176519fb0224\") "
Oct 30 20:12:53.243385 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:53.243314    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25efa968-ddb4-4893-8209-176519fb0224-openshift-service-ca" pod "25efa968-ddb4-4893-8209-176519fb0224" (UID: "25efa968-ddb4-4893-8209-176519fb0224"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:53.243751 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:53.243731    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25efa968-ddb4-4893-8209-176519fb0224-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:53.244920 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:53.244893    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25efa968-ddb4-4893-8209-176519fb0224-maas-api-tls" pod "25efa968-ddb4-4893-8209-176519fb0224" (UID: "25efa968-ddb4-4893-8209-176519fb0224"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:53.245171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:53.245152    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25efa968-ddb4-4893-8209-176519fb0224-kube-api-access-2vxqw" pod "25efa968-ddb4-4893-8209-176519fb0224" (UID: "25efa968-ddb4-4893-8209-176519fb0224"). InnerVolumeSpecName "kube-api-access-2vxqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:53.245459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:53.245434    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25efa968-ddb4-4893-8209-176519fb0224-metrics-certs" pod "25efa968-ddb4-4893-8209-176519fb0224" (UID: "25efa968-ddb4-4893-8209-176519fb0224"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:53.345719 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:53.345622    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vxqw\" (UniqueName: \"kubernetes.io/projected/25efa968-ddb4-4893-8209-176519fb0224-kube-api-access-2vxqw\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:53.345719 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:53.345657    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25efa968-ddb4-4893-8209-176519fb0224-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:53.345719 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:53.345671    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25efa968-ddb4-4893-8209-176519fb0224-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:54.158331 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:54.158302    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"
Oct 30 20:12:54.158747 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:54.158302    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8" event={"ID":"25efa968-ddb4-4893-8209-176519fb0224","Type":"ContainerDied","Data":"1d0ea7fd5e14f9cb29c58dece16a42d5d8432e9758e5ee113338f5720ad660bf"}
Oct 30 20:12:54.158747 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:54.158436    5408 scope.go:122] "RemoveContainer" containerID="402d598b56ba3b38e165b41c30b8e8f9d111fde41e99a3784619b59bd57e9b2f"
Oct 30 20:12:54.198890 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:54.198855    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"]
Oct 30 20:12:54.211170 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:54.211151    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3f57c5-6f568f769b-r9xw8"]
Oct 30 20:12:55.531680 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:55.531650    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25efa968-ddb4-4893-8209-176519fb0224" path="/var/lib/kubelet/pods/25efa968-ddb4-4893-8209-176519fb0224/volumes"
Oct 30 20:12:58.234306 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.234206    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" containerID="cri-o://960674a42a3f997671a402c0590e536b66f2b0f9077219cd44bfbab437290b2a" gracePeriod=45
Oct 30 20:12:58.538838 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.538811    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 20:12:58.603120 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.603087    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-tmp-dir\") pod \"920fc884-8d5e-40e5-9e67-550521b12e3e\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") "
Oct 30 20:12:58.603309 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.603152    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/920fc884-8d5e-40e5-9e67-550521b12e3e-tls-certs\" (UniqueName: \"kubernetes.io/secret/920fc884-8d5e-40e5-9e67-550521b12e3e-tls-certs\") pod \"920fc884-8d5e-40e5-9e67-550521b12e3e\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") "
Oct 30 20:12:58.603309 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.603216    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/920fc884-8d5e-40e5-9e67-550521b12e3e-kube-api-access-kwk5w\" (UniqueName: \"kubernetes.io/projected/920fc884-8d5e-40e5-9e67-550521b12e3e-kube-api-access-kwk5w\") pod \"920fc884-8d5e-40e5-9e67-550521b12e3e\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") "
Oct 30 20:12:58.603309 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.603289    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-home\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-home\") pod \"920fc884-8d5e-40e5-9e67-550521b12e3e\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") "
Oct 30 20:12:58.603502 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.603342    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-dshm\") pod \"920fc884-8d5e-40e5-9e67-550521b12e3e\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") "
Oct 30 20:12:58.603502 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.603363    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-tmp-dir" pod "920fc884-8d5e-40e5-9e67-550521b12e3e" (UID: "920fc884-8d5e-40e5-9e67-550521b12e3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:58.603502 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.603425    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-model-cache\") pod \"920fc884-8d5e-40e5-9e67-550521b12e3e\" (UID: \"920fc884-8d5e-40e5-9e67-550521b12e3e\") "
Oct 30 20:12:58.603682 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.603607    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-home" pod "920fc884-8d5e-40e5-9e67-550521b12e3e" (UID: "920fc884-8d5e-40e5-9e67-550521b12e3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:58.603852 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.603828    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-model-cache" pod "920fc884-8d5e-40e5-9e67-550521b12e3e" (UID: "920fc884-8d5e-40e5-9e67-550521b12e3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:58.603974 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.603903    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:58.603974 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.603922    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:58.606013 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.605986    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920fc884-8d5e-40e5-9e67-550521b12e3e-tls-certs" pod "920fc884-8d5e-40e5-9e67-550521b12e3e" (UID: "920fc884-8d5e-40e5-9e67-550521b12e3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:58.606221 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.606021    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920fc884-8d5e-40e5-9e67-550521b12e3e-kube-api-access-kwk5w" pod "920fc884-8d5e-40e5-9e67-550521b12e3e" (UID: "920fc884-8d5e-40e5-9e67-550521b12e3e"). InnerVolumeSpecName "kube-api-access-kwk5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:58.606498 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.606474    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-dshm" pod "920fc884-8d5e-40e5-9e67-550521b12e3e" (UID: "920fc884-8d5e-40e5-9e67-550521b12e3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:58.704882 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.704852    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920fc884-8d5e-40e5-9e67-550521b12e3e-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:58.704882 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.704880    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwk5w\" (UniqueName: \"kubernetes.io/projected/920fc884-8d5e-40e5-9e67-550521b12e3e-kube-api-access-kwk5w\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:58.704882 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.704889    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:58.704882 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:58.704898    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920fc884-8d5e-40e5-9e67-550521b12e3e-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:59.232317 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:59.232283    5408 generic.go:356] "Generic (PLEG): container finished" podID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerID="960674a42a3f997671a402c0590e536b66f2b0f9077219cd44bfbab437290b2a" exitCode=0
Oct 30 20:12:59.232528 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:59.232326    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"
Oct 30 20:12:59.232528 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:59.232365    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" event={"ID":"920fc884-8d5e-40e5-9e67-550521b12e3e","Type":"ContainerDied","Data":"960674a42a3f997671a402c0590e536b66f2b0f9077219cd44bfbab437290b2a"}
Oct 30 20:12:59.232528 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:59.232410    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" event={"ID":"920fc884-8d5e-40e5-9e67-550521b12e3e","Type":"ContainerDied","Data":"a21f64cf225bb1f46cfbfb541303e1ce958daaeed2efadaf88e8d61b58942aab"}
Oct 30 20:12:59.232528 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:59.232430    5408 scope.go:122] "RemoveContainer" containerID="960674a42a3f997671a402c0590e536b66f2b0f9077219cd44bfbab437290b2a"
Oct 30 20:12:59.242272 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:59.242072    5408 scope.go:122] "RemoveContainer" containerID="960674a42a3f997671a402c0590e536b66f2b0f9077219cd44bfbab437290b2a"
Oct 30 20:12:59.242510 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:12:59.242338    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960674a42a3f997671a402c0590e536b66f2b0f9077219cd44bfbab437290b2a\": container with ID starting with 960674a42a3f997671a402c0590e536b66f2b0f9077219cd44bfbab437290b2a not found: ID does not exist" containerID="960674a42a3f997671a402c0590e536b66f2b0f9077219cd44bfbab437290b2a"
Oct 30 20:12:59.242510 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:59.242371    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960674a42a3f997671a402c0590e536b66f2b0f9077219cd44bfbab437290b2a"} err="failed to get container status \"960674a42a3f997671a402c0590e536b66f2b0f9077219cd44bfbab437290b2a\": rpc error: code = NotFound desc = could not find container \"960674a42a3f997671a402c0590e536b66f2b0f9077219cd44bfbab437290b2a\": container with ID starting with 960674a42a3f997671a402c0590e536b66f2b0f9077219cd44bfbab437290b2a not found: ID does not exist"
Oct 30 20:12:59.287658 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:59.287632    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"]
Oct 30 20:12:59.297205 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:59.297183    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb"]
Oct 30 20:12:59.502025 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:59.501964    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-2-w6-5f933e-kserve-f9448484c-rlgnb" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:12:59.532164 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:12:59.532137    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" path="/var/lib/kubelet/pods/920fc884-8d5e-40e5-9e67-550521b12e3e/volumes"
Oct 30 20:13:07.765539 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:07.765504    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"]
Oct 30 20:13:07.766048 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:07.765707    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy" containerID="cri-o://7bfb4ea8152e81b8a525bc6c87d2bc87c89ddf059095af814485d668e6c4f981" gracePeriod=30
Oct 30 20:13:07.783532 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:07.783499    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"]
Oct 30 20:13:07.783757 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:07.783732    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy" containerID="cri-o://d9cfd4bbda33519067b37f3ca65567fabdac4388908769106429df467c2134e4" gracePeriod=30
Oct 30 20:13:07.809075 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:07.804192    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"]
Oct 30 20:13:07.809075 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:07.804414    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb" podUID="6c2a0744-9e32-4896-914e-dfb9173923e7" containerName="istio-proxy" containerID="cri-o://da9f1f5f7532d452ee472e50791f1e02e3637f6af2292d06fedbd9a3834fbb31" gracePeriod=30
Oct 30 20:13:08.518641 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:08.518598    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:13:08.518802 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:08.518670    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb" podUID="6c2a0744-9e32-4896-914e-dfb9173923e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:13:09.662070 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:13:09.662021    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:13:09.821135 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:09.821096    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"]
Oct 30 20:13:09.821402 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:09.821374    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm" podUID="75115e13-a71f-49ff-8bbc-6df7eebd4195" containerName="istio-proxy" containerID="cri-o://6754b42b0f5681bf1388824942b0e8736f59e6217296b7acbc2c780254e55d07" gracePeriod=30
Oct 30 20:13:09.822487 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:09.822466    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"]
Oct 30 20:13:09.822660 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:09.822632    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84" podUID="a8c73d1b-a864-4534-944a-a543d04a2c3b" containerName="istio-proxy" containerID="cri-o://476f128b0efd88bdb29619b29dc2a41c8ed9544f81491afd76a1430d130eee24" gracePeriod=30
Oct 30 20:13:11.127541 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:11.127507    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"]
Oct 30 20:13:11.128063 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:11.127726    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp" podUID="7577f068-6466-4add-8edf-691d113e8eb2" containerName="istio-proxy" containerID="cri-o://7301acfcb0c3035d3f5a59ff56d0a539d59054eb24a09e62efcc35ebc78912a8" gracePeriod=30
Oct 30 20:13:11.144560 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:11.144525    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"]
Oct 30 20:13:11.144774 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:11.144751    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt" podUID="bdae6413-f5be-452b-a5a7-d6ca4abb9081" containerName="istio-proxy" containerID="cri-o://9f27cdc03e3ec7fcc279e3f97fd84dde90b340224211a383ad72e68cb975209e" gracePeriod=30
Oct 30 20:13:11.307459 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:11.307422    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:13:11.307730 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:11.307698    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:13:12.158944 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:12.158901    5408 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:13:12.159385 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:12.158963    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:13:12.564298 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:12.564262    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"]
Oct 30 20:13:12.564568 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:12.564478    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj" podUID="5b518877-72bf-4743-8f2e-daa5f9800392" containerName="maas-api" containerID="cri-o://7f950bd8970eac766cc35fdcfc09df95aea64da2e84931504bca9f8ff0861ccc" gracePeriod=30
Oct 30 20:13:12.986121 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:12.985380    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"
Oct 30 20:13:13.152980 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.152941    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b518877-72bf-4743-8f2e-daa5f9800392-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b518877-72bf-4743-8f2e-daa5f9800392-openshift-service-ca\") pod \"5b518877-72bf-4743-8f2e-daa5f9800392\" (UID: \"5b518877-72bf-4743-8f2e-daa5f9800392\") "
Oct 30 20:13:13.153184 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.153165    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b518877-72bf-4743-8f2e-daa5f9800392-kube-api-access-vx4sb\" (UniqueName: \"kubernetes.io/projected/5b518877-72bf-4743-8f2e-daa5f9800392-kube-api-access-vx4sb\") pod \"5b518877-72bf-4743-8f2e-daa5f9800392\" (UID: \"5b518877-72bf-4743-8f2e-daa5f9800392\") "
Oct 30 20:13:13.153249 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.153210    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b518877-72bf-4743-8f2e-daa5f9800392-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b518877-72bf-4743-8f2e-daa5f9800392-maas-api-tls\") pod \"5b518877-72bf-4743-8f2e-daa5f9800392\" (UID: \"5b518877-72bf-4743-8f2e-daa5f9800392\") "
Oct 30 20:13:13.153316 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.153280    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b518877-72bf-4743-8f2e-daa5f9800392-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b518877-72bf-4743-8f2e-daa5f9800392-metrics-certs\") pod \"5b518877-72bf-4743-8f2e-daa5f9800392\" (UID: \"5b518877-72bf-4743-8f2e-daa5f9800392\") "
Oct 30 20:13:13.154423 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.154393    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b518877-72bf-4743-8f2e-daa5f9800392-openshift-service-ca" pod "5b518877-72bf-4743-8f2e-daa5f9800392" (UID: "5b518877-72bf-4743-8f2e-daa5f9800392"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:13.156760 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.156731    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b518877-72bf-4743-8f2e-daa5f9800392-kube-api-access-vx4sb" pod "5b518877-72bf-4743-8f2e-daa5f9800392" (UID: "5b518877-72bf-4743-8f2e-daa5f9800392"). InnerVolumeSpecName "kube-api-access-vx4sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:13.157022 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.156881    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b518877-72bf-4743-8f2e-daa5f9800392-maas-api-tls" pod "5b518877-72bf-4743-8f2e-daa5f9800392" (UID: "5b518877-72bf-4743-8f2e-daa5f9800392"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:13.158711 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.158683    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b518877-72bf-4743-8f2e-daa5f9800392-metrics-certs" pod "5b518877-72bf-4743-8f2e-daa5f9800392" (UID: "5b518877-72bf-4743-8f2e-daa5f9800392"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:13.174520 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.174497    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 20:13:13.249856 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.249827    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 20:13:13.252865 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.252843    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 20:13:13.254466 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.254443    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-data\") pod \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") "
Oct 30 20:13:13.254571 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.254493    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istiod-ca-cert\") pod \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") "
Oct 30 20:13:13.254571 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.254535    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-token\" (UniqueName: \"kubernetes.io/projected/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-token\") pod \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") "
Oct 30 20:13:13.254647 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.254586    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-podinfo\") pod \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") "
Oct 30 20:13:13.254647 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.254611    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-credential-socket\") pod \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") "
Oct 30 20:13:13.254647 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.254630    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-workload-certs\") pod \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") "
Oct 30 20:13:13.254784 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.254656    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3086a82d-b9ca-4c66-8b65-a1eb764909e5-kube-api-access-bhd6n\" (UniqueName: \"kubernetes.io/projected/3086a82d-b9ca-4c66-8b65-a1eb764909e5-kube-api-access-bhd6n\") pod \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") "
Oct 30 20:13:13.254784 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.254714    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-envoy\") pod \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") "
Oct 30 20:13:13.258069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.255319    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-credential-socket" pod "3086a82d-b9ca-4c66-8b65-a1eb764909e5" (UID: "3086a82d-b9ca-4c66-8b65-a1eb764909e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.258069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.255901    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istiod-ca-cert" pod "3086a82d-b9ca-4c66-8b65-a1eb764909e5" (UID: "3086a82d-b9ca-4c66-8b65-a1eb764909e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:13.258069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.256125    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-workload-socket\") pod \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\" (UID: \"3086a82d-b9ca-4c66-8b65-a1eb764909e5\") "
Oct 30 20:13:13.258069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.256158    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-data" pod "3086a82d-b9ca-4c66-8b65-a1eb764909e5" (UID: "3086a82d-b9ca-4c66-8b65-a1eb764909e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.258069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.256416    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-workload-socket" pod "3086a82d-b9ca-4c66-8b65-a1eb764909e5" (UID: "3086a82d-b9ca-4c66-8b65-a1eb764909e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.258069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.256855    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.258069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.256880    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b518877-72bf-4743-8f2e-daa5f9800392-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.258069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.256896    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.258069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.256910    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.258069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.256923    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx4sb\" (UniqueName: \"kubernetes.io/projected/5b518877-72bf-4743-8f2e-daa5f9800392-kube-api-access-vx4sb\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.258069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.256938    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b518877-72bf-4743-8f2e-daa5f9800392-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.258069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.256951    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b518877-72bf-4743-8f2e-daa5f9800392-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.258069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.256968    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.258069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.257211    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-workload-certs" pod "3086a82d-b9ca-4c66-8b65-a1eb764909e5" (UID: "3086a82d-b9ca-4c66-8b65-a1eb764909e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.258069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.257412    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-token" pod "3086a82d-b9ca-4c66-8b65-a1eb764909e5" (UID: "3086a82d-b9ca-4c66-8b65-a1eb764909e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:13.258069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.257998    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3086a82d-b9ca-4c66-8b65-a1eb764909e5-kube-api-access-bhd6n" pod "3086a82d-b9ca-4c66-8b65-a1eb764909e5" (UID: "3086a82d-b9ca-4c66-8b65-a1eb764909e5"). InnerVolumeSpecName "kube-api-access-bhd6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:13.258643 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.258468    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-podinfo" pod "3086a82d-b9ca-4c66-8b65-a1eb764909e5" (UID: "3086a82d-b9ca-4c66-8b65-a1eb764909e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:13:13.263071 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.258851    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-envoy" pod "3086a82d-b9ca-4c66-8b65-a1eb764909e5" (UID: "3086a82d-b9ca-4c66-8b65-a1eb764909e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.333164 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.333127    5408 generic.go:356] "Generic (PLEG): container finished" podID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerID="d9cfd4bbda33519067b37f3ca65567fabdac4388908769106429df467c2134e4" exitCode=0
Oct 30 20:13:13.333360 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.333188    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"
Oct 30 20:13:13.333360 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.333210    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" event={"ID":"3086a82d-b9ca-4c66-8b65-a1eb764909e5","Type":"ContainerDied","Data":"d9cfd4bbda33519067b37f3ca65567fabdac4388908769106429df467c2134e4"}
Oct 30 20:13:13.333360 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.333299    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h" event={"ID":"3086a82d-b9ca-4c66-8b65-a1eb764909e5","Type":"ContainerDied","Data":"eb495302a0cc801b6f90761d0e74f0489c3f339019faabef03ffaf64adecddd4"}
Oct 30 20:13:13.333360 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.333336    5408 scope.go:122] "RemoveContainer" containerID="d9cfd4bbda33519067b37f3ca65567fabdac4388908769106429df467c2134e4"
Oct 30 20:13:13.335263 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.335210    5408 generic.go:356] "Generic (PLEG): container finished" podID="5b518877-72bf-4743-8f2e-daa5f9800392" containerID="7f950bd8970eac766cc35fdcfc09df95aea64da2e84931504bca9f8ff0861ccc" exitCode=0
Oct 30 20:13:13.335392 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.335354    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj" event={"ID":"5b518877-72bf-4743-8f2e-daa5f9800392","Type":"ContainerDied","Data":"7f950bd8970eac766cc35fdcfc09df95aea64da2e84931504bca9f8ff0861ccc"}
Oct 30 20:13:13.335464 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.335393    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj" event={"ID":"5b518877-72bf-4743-8f2e-daa5f9800392","Type":"ContainerDied","Data":"d94a8e99192ca159b14aec6eebe481fc1c4181c097c89806613cbce13b3b0932"}
Oct 30 20:13:13.335524 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.335482    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"
Oct 30 20:13:13.345944 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.345927    5408 scope.go:122] "RemoveContainer" containerID="d9cfd4bbda33519067b37f3ca65567fabdac4388908769106429df467c2134e4"
Oct 30 20:13:13.346083 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.345965    5408 generic.go:356] "Generic (PLEG): container finished" podID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerID="7bfb4ea8152e81b8a525bc6c87d2bc87c89ddf059095af814485d668e6c4f981" exitCode=0
Oct 30 20:13:13.346083 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.346078    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"
Oct 30 20:13:13.346230 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.346074    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" event={"ID":"d7619cb9-6eef-4207-a4ee-a892f1920ff6","Type":"ContainerDied","Data":"7bfb4ea8152e81b8a525bc6c87d2bc87c89ddf059095af814485d668e6c4f981"}
Oct 30 20:13:13.346230 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.346200    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l" event={"ID":"d7619cb9-6eef-4207-a4ee-a892f1920ff6","Type":"ContainerDied","Data":"a2ada82e4a18843045c06dc6ef0f7c69e2d30ed4aeeb7272fbfa64740e497406"}
Oct 30 20:13:13.346607 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:13:13.346579    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9cfd4bbda33519067b37f3ca65567fabdac4388908769106429df467c2134e4\": container with ID starting with d9cfd4bbda33519067b37f3ca65567fabdac4388908769106429df467c2134e4 not found: ID does not exist" containerID="d9cfd4bbda33519067b37f3ca65567fabdac4388908769106429df467c2134e4"
Oct 30 20:13:13.346718 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.346617    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9cfd4bbda33519067b37f3ca65567fabdac4388908769106429df467c2134e4"} err="failed to get container status \"d9cfd4bbda33519067b37f3ca65567fabdac4388908769106429df467c2134e4\": rpc error: code = NotFound desc = could not find container \"d9cfd4bbda33519067b37f3ca65567fabdac4388908769106429df467c2134e4\": container with ID starting with d9cfd4bbda33519067b37f3ca65567fabdac4388908769106429df467c2134e4 not found: ID does not exist"
Oct 30 20:13:13.346718 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.346640    5408 scope.go:122] "RemoveContainer" containerID="7f950bd8970eac766cc35fdcfc09df95aea64da2e84931504bca9f8ff0861ccc"
Oct 30 20:13:13.348132 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.348111    5408 generic.go:356] "Generic (PLEG): container finished" podID="6c2a0744-9e32-4896-914e-dfb9173923e7" containerID="da9f1f5f7532d452ee472e50791f1e02e3637f6af2292d06fedbd9a3834fbb31" exitCode=0
Oct 30 20:13:13.348328 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.348155    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"
Oct 30 20:13:13.348328 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.348189    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb" event={"ID":"6c2a0744-9e32-4896-914e-dfb9173923e7","Type":"ContainerDied","Data":"da9f1f5f7532d452ee472e50791f1e02e3637f6af2292d06fedbd9a3834fbb31"}
Oct 30 20:13:13.348328 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.348229    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb" event={"ID":"6c2a0744-9e32-4896-914e-dfb9173923e7","Type":"ContainerDied","Data":"95c635a31a768de95686f19c6b6c119980b0c6d4ced212b72d331c629ea85bf2"}
Oct 30 20:13:13.355792 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.355775    5408 scope.go:122] "RemoveContainer" containerID="7f950bd8970eac766cc35fdcfc09df95aea64da2e84931504bca9f8ff0861ccc"
Oct 30 20:13:13.356118 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:13:13.356093    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f950bd8970eac766cc35fdcfc09df95aea64da2e84931504bca9f8ff0861ccc\": container with ID starting with 7f950bd8970eac766cc35fdcfc09df95aea64da2e84931504bca9f8ff0861ccc not found: ID does not exist" containerID="7f950bd8970eac766cc35fdcfc09df95aea64da2e84931504bca9f8ff0861ccc"
Oct 30 20:13:13.356211 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.356124    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f950bd8970eac766cc35fdcfc09df95aea64da2e84931504bca9f8ff0861ccc"} err="failed to get container status \"7f950bd8970eac766cc35fdcfc09df95aea64da2e84931504bca9f8ff0861ccc\": rpc error: code = NotFound desc = could not find container \"7f950bd8970eac766cc35fdcfc09df95aea64da2e84931504bca9f8ff0861ccc\": container with ID starting with 7f950bd8970eac766cc35fdcfc09df95aea64da2e84931504bca9f8ff0861ccc not found: ID does not exist"
Oct 30 20:13:13.356211 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.356139    5408 scope.go:122] "RemoveContainer" containerID="7bfb4ea8152e81b8a525bc6c87d2bc87c89ddf059095af814485d668e6c4f981"
Oct 30 20:13:13.358260 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358241    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-envoy\") pod \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") "
Oct 30 20:13:13.358357 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358280    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-token\" (UniqueName: \"kubernetes.io/projected/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-token\") pod \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") "
Oct 30 20:13:13.358357 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358319    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7619cb9-6eef-4207-a4ee-a892f1920ff6-kube-api-access-d7sww\" (UniqueName: \"kubernetes.io/projected/d7619cb9-6eef-4207-a4ee-a892f1920ff6-kube-api-access-d7sww\") pod \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") "
Oct 30 20:13:13.358357 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358357    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c2a0744-9e32-4896-914e-dfb9173923e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2a0744-9e32-4896-914e-dfb9173923e7-istiod-ca-cert\") pod \"6c2a0744-9e32-4896-914e-dfb9173923e7\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") "
Oct 30 20:13:13.358487 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358442    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-workload-certs\") pod \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") "
Oct 30 20:13:13.358544 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358485    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-token\" (UniqueName: \"kubernetes.io/projected/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-token\") pod \"6c2a0744-9e32-4896-914e-dfb9173923e7\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") "
Oct 30 20:13:13.358544 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358513    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-workload-certs\") pod \"6c2a0744-9e32-4896-914e-dfb9173923e7\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") "
Oct 30 20:13:13.358665 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358540    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-envoy\") pod \"6c2a0744-9e32-4896-914e-dfb9173923e7\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") "
Oct 30 20:13:13.358665 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358601    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-workload-socket\") pod \"6c2a0744-9e32-4896-914e-dfb9173923e7\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") "
Oct 30 20:13:13.358775 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358663    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-podinfo\") pod \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") "
Oct 30 20:13:13.358775 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358697    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-credential-socket\") pod \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") "
Oct 30 20:13:13.358775 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358732    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-podinfo\") pod \"6c2a0744-9e32-4896-914e-dfb9173923e7\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") "
Oct 30 20:13:13.358775 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358766    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-workload-socket\") pod \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") "
Oct 30 20:13:13.359002 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358818    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2a0744-9e32-4896-914e-dfb9173923e7-kube-api-access-pcj94\" (UniqueName: \"kubernetes.io/projected/6c2a0744-9e32-4896-914e-dfb9173923e7-kube-api-access-pcj94\") pod \"6c2a0744-9e32-4896-914e-dfb9173923e7\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") "
Oct 30 20:13:13.359002 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358853    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-data\") pod \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") "
Oct 30 20:13:13.359002 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358907    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-data\") pod \"6c2a0744-9e32-4896-914e-dfb9173923e7\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") "
Oct 30 20:13:13.359002 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358939    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-credential-socket\") pod \"6c2a0744-9e32-4896-914e-dfb9173923e7\" (UID: \"6c2a0744-9e32-4896-914e-dfb9173923e7\") "
Oct 30 20:13:13.359002 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358943    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c2a0744-9e32-4896-914e-dfb9173923e7-istiod-ca-cert" pod "6c2a0744-9e32-4896-914e-dfb9173923e7" (UID: "6c2a0744-9e32-4896-914e-dfb9173923e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:13.359002 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.358989    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istiod-ca-cert\") pod \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\" (UID: \"d7619cb9-6eef-4207-a4ee-a892f1920ff6\") "
Oct 30 20:13:13.359344 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.359251    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-workload-certs" pod "6c2a0744-9e32-4896-914e-dfb9173923e7" (UID: "6c2a0744-9e32-4896-914e-dfb9173923e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.359539 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.359519    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istiod-ca-cert" pod "d7619cb9-6eef-4207-a4ee-a892f1920ff6" (UID: "d7619cb9-6eef-4207-a4ee-a892f1920ff6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:13.359539 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.359522    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.359733 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.359556    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.359733 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.359574    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2a0744-9e32-4896-914e-dfb9173923e7-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.359733 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.359588    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3086a82d-b9ca-4c66-8b65-a1eb764909e5-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.359733 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.359602    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3086a82d-b9ca-4c66-8b65-a1eb764909e5-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.359733 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.359617    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhd6n\" (UniqueName: \"kubernetes.io/projected/3086a82d-b9ca-4c66-8b65-a1eb764909e5-kube-api-access-bhd6n\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.359733 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.359630    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.359733 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.359728    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-workload-certs" pod "d7619cb9-6eef-4207-a4ee-a892f1920ff6" (UID: "d7619cb9-6eef-4207-a4ee-a892f1920ff6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.360304 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.359874    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-credential-socket" pod "d7619cb9-6eef-4207-a4ee-a892f1920ff6" (UID: "d7619cb9-6eef-4207-a4ee-a892f1920ff6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.368581 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.368429    5408 scope.go:122] "RemoveContainer" containerID="7bfb4ea8152e81b8a525bc6c87d2bc87c89ddf059095af814485d668e6c4f981"
Oct 30 20:13:13.369055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.368980    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-data" pod "d7619cb9-6eef-4207-a4ee-a892f1920ff6" (UID: "d7619cb9-6eef-4207-a4ee-a892f1920ff6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.369156 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.369002    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-workload-socket" pod "6c2a0744-9e32-4896-914e-dfb9173923e7" (UID: "6c2a0744-9e32-4896-914e-dfb9173923e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.369213 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:13:13.369099    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bfb4ea8152e81b8a525bc6c87d2bc87c89ddf059095af814485d668e6c4f981\": container with ID starting with 7bfb4ea8152e81b8a525bc6c87d2bc87c89ddf059095af814485d668e6c4f981 not found: ID does not exist" containerID="7bfb4ea8152e81b8a525bc6c87d2bc87c89ddf059095af814485d668e6c4f981"
Oct 30 20:13:13.369213 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.369157    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-workload-socket" pod "d7619cb9-6eef-4207-a4ee-a892f1920ff6" (UID: "d7619cb9-6eef-4207-a4ee-a892f1920ff6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.369213 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.369192    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bfb4ea8152e81b8a525bc6c87d2bc87c89ddf059095af814485d668e6c4f981"} err="failed to get container status \"7bfb4ea8152e81b8a525bc6c87d2bc87c89ddf059095af814485d668e6c4f981\": rpc error: code = NotFound desc = could not find container \"7bfb4ea8152e81b8a525bc6c87d2bc87c89ddf059095af814485d668e6c4f981\": container with ID starting with 7bfb4ea8152e81b8a525bc6c87d2bc87c89ddf059095af814485d668e6c4f981 not found: ID does not exist"
Oct 30 20:13:13.369378 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.369222    5408 scope.go:122] "RemoveContainer" containerID="da9f1f5f7532d452ee472e50791f1e02e3637f6af2292d06fedbd9a3834fbb31"
Oct 30 20:13:13.369378 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.369196    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-credential-socket" pod "6c2a0744-9e32-4896-914e-dfb9173923e7" (UID: "6c2a0744-9e32-4896-914e-dfb9173923e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.369490 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.369376    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-data" pod "6c2a0744-9e32-4896-914e-dfb9173923e7" (UID: "6c2a0744-9e32-4896-914e-dfb9173923e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.371208 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.371182    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2a0744-9e32-4896-914e-dfb9173923e7-kube-api-access-pcj94" pod "6c2a0744-9e32-4896-914e-dfb9173923e7" (UID: "6c2a0744-9e32-4896-914e-dfb9173923e7"). InnerVolumeSpecName "kube-api-access-pcj94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:13.371342 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.371322    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-token" pod "d7619cb9-6eef-4207-a4ee-a892f1920ff6" (UID: "d7619cb9-6eef-4207-a4ee-a892f1920ff6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:13.371630 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.371609    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"]
Oct 30 20:13:13.373519 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.373497    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rtk5h"]
Oct 30 20:13:13.377786 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.377770    5408 scope.go:122] "RemoveContainer" containerID="da9f1f5f7532d452ee472e50791f1e02e3637f6af2292d06fedbd9a3834fbb31"
Oct 30 20:13:13.378051 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:13:13.378019    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da9f1f5f7532d452ee472e50791f1e02e3637f6af2292d06fedbd9a3834fbb31\": container with ID starting with da9f1f5f7532d452ee472e50791f1e02e3637f6af2292d06fedbd9a3834fbb31 not found: ID does not exist" containerID="da9f1f5f7532d452ee472e50791f1e02e3637f6af2292d06fedbd9a3834fbb31"
Oct 30 20:13:13.378113 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.378057    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da9f1f5f7532d452ee472e50791f1e02e3637f6af2292d06fedbd9a3834fbb31"} err="failed to get container status \"da9f1f5f7532d452ee472e50791f1e02e3637f6af2292d06fedbd9a3834fbb31\": rpc error: code = NotFound desc = could not find container \"da9f1f5f7532d452ee472e50791f1e02e3637f6af2292d06fedbd9a3834fbb31\": container with ID starting with da9f1f5f7532d452ee472e50791f1e02e3637f6af2292d06fedbd9a3834fbb31 not found: ID does not exist"
Oct 30 20:13:13.382536 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.382442    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-envoy" pod "d7619cb9-6eef-4207-a4ee-a892f1920ff6" (UID: "d7619cb9-6eef-4207-a4ee-a892f1920ff6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.382536 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.382444    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-podinfo" pod "6c2a0744-9e32-4896-914e-dfb9173923e7" (UID: "6c2a0744-9e32-4896-914e-dfb9173923e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:13:13.382536 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.382493    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-envoy" pod "6c2a0744-9e32-4896-914e-dfb9173923e7" (UID: "6c2a0744-9e32-4896-914e-dfb9173923e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.382536 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.382513    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7619cb9-6eef-4207-a4ee-a892f1920ff6-kube-api-access-d7sww" pod "d7619cb9-6eef-4207-a4ee-a892f1920ff6" (UID: "d7619cb9-6eef-4207-a4ee-a892f1920ff6"). InnerVolumeSpecName "kube-api-access-d7sww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:13.382536 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.382514    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-podinfo" pod "d7619cb9-6eef-4207-a4ee-a892f1920ff6" (UID: "d7619cb9-6eef-4207-a4ee-a892f1920ff6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:13:13.382536 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.382534    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-token" pod "6c2a0744-9e32-4896-914e-dfb9173923e7" (UID: "6c2a0744-9e32-4896-914e-dfb9173923e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:13.399520 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.399498    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"]
Oct 30 20:13:13.401596 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.401573    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5f933e-6c6c55c65b-vfvsj"]
Oct 30 20:13:13.460469 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.460446    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.460469 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.460470    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.460607 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.460481    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7sww\" (UniqueName: \"kubernetes.io/projected/d7619cb9-6eef-4207-a4ee-a892f1920ff6-kube-api-access-d7sww\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.460607 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.460492    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.460607 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.460501    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.460607 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.460511    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.460607 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.460520    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.460607 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.460528    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.460607 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.460536    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.460607 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.460545    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.460607 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.460553    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.460607 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.460562    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcj94\" (UniqueName: \"kubernetes.io/projected/6c2a0744-9e32-4896-914e-dfb9173923e7-kube-api-access-pcj94\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.460607 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.460570    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.460607 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.460578    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.460607 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.460587    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a0744-9e32-4896-914e-dfb9173923e7-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.460607 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.460594    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7619cb9-6eef-4207-a4ee-a892f1920ff6-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.532767 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.532739    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" path="/var/lib/kubelet/pods/3086a82d-b9ca-4c66-8b65-a1eb764909e5/volumes"
Oct 30 20:13:13.533209 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.533194    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b518877-72bf-4743-8f2e-daa5f9800392" path="/var/lib/kubelet/pods/5b518877-72bf-4743-8f2e-daa5f9800392/volumes"
Oct 30 20:13:13.668195 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.668118    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"]
Oct 30 20:13:13.674065 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.674016    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-h4n5l"]
Oct 30 20:13:13.686568 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.686543    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"]
Oct 30 20:13:13.693715 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.693693    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-8msgb"]
Oct 30 20:13:13.732703 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.732676    5408 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:13:13.732800 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.732727    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt" podUID="bdae6413-f5be-452b-a5a7-d6ca4abb9081" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:13:13.866543 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.866510    5408 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:13:13.866684 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:13.866574    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp" podUID="7577f068-6466-4add-8edf-691d113e8eb2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:13:15.101364 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.101326    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 20:13:15.104361 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.104341    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 20:13:15.278900 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.278867    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-envoy\") pod \"a8c73d1b-a864-4534-944a-a543d04a2c3b\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") "
Oct 30 20:13:15.279106 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.278917    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c73d1b-a864-4534-944a-a543d04a2c3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c73d1b-a864-4534-944a-a543d04a2c3b-istiod-ca-cert\") pod \"a8c73d1b-a864-4534-944a-a543d04a2c3b\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") "
Oct 30 20:13:15.279106 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.278949    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-token\" (UniqueName: \"kubernetes.io/projected/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-token\") pod \"a8c73d1b-a864-4534-944a-a543d04a2c3b\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") "
Oct 30 20:13:15.279106 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.278995    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-data\") pod \"75115e13-a71f-49ff-8bbc-6df7eebd4195\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") "
Oct 30 20:13:15.279106 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279050    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-token\" (UniqueName: \"kubernetes.io/projected/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-token\") pod \"75115e13-a71f-49ff-8bbc-6df7eebd4195\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") "
Oct 30 20:13:15.279106 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279083    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-workload-socket\") pod \"75115e13-a71f-49ff-8bbc-6df7eebd4195\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") "
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279114    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75115e13-a71f-49ff-8bbc-6df7eebd4195-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75115e13-a71f-49ff-8bbc-6df7eebd4195-istiod-ca-cert\") pod \"75115e13-a71f-49ff-8bbc-6df7eebd4195\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") "
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279154    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-data\") pod \"a8c73d1b-a864-4534-944a-a543d04a2c3b\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") "
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279197    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75115e13-a71f-49ff-8bbc-6df7eebd4195-kube-api-access-2wp2h\" (UniqueName: \"kubernetes.io/projected/75115e13-a71f-49ff-8bbc-6df7eebd4195-kube-api-access-2wp2h\") pod \"75115e13-a71f-49ff-8bbc-6df7eebd4195\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") "
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279265    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-workload-socket\") pod \"a8c73d1b-a864-4534-944a-a543d04a2c3b\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") "
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279320    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-podinfo\") pod \"75115e13-a71f-49ff-8bbc-6df7eebd4195\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") "
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279349    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-credential-socket\") pod \"a8c73d1b-a864-4534-944a-a543d04a2c3b\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") "
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279389    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-credential-socket\") pod \"75115e13-a71f-49ff-8bbc-6df7eebd4195\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") "
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279420    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-envoy\") pod \"75115e13-a71f-49ff-8bbc-6df7eebd4195\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") "
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279435    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c73d1b-a864-4534-944a-a543d04a2c3b-istiod-ca-cert" pod "a8c73d1b-a864-4534-944a-a543d04a2c3b" (UID: "a8c73d1b-a864-4534-944a-a543d04a2c3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279513    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-workload-certs\") pod \"75115e13-a71f-49ff-8bbc-6df7eebd4195\" (UID: \"75115e13-a71f-49ff-8bbc-6df7eebd4195\") "
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279542    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c73d1b-a864-4534-944a-a543d04a2c3b-kube-api-access-7kp6t\" (UniqueName: \"kubernetes.io/projected/a8c73d1b-a864-4534-944a-a543d04a2c3b-kube-api-access-7kp6t\") pod \"a8c73d1b-a864-4534-944a-a543d04a2c3b\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") "
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279592    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-podinfo\") pod \"a8c73d1b-a864-4534-944a-a543d04a2c3b\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") "
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279637    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-data" pod "75115e13-a71f-49ff-8bbc-6df7eebd4195" (UID: "75115e13-a71f-49ff-8bbc-6df7eebd4195"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279671    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-workload-certs\") pod \"a8c73d1b-a864-4534-944a-a543d04a2c3b\" (UID: \"a8c73d1b-a864-4534-944a-a543d04a2c3b\") "
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279714    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-credential-socket" pod "a8c73d1b-a864-4534-944a-a543d04a2c3b" (UID: "a8c73d1b-a864-4534-944a-a543d04a2c3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279766    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-data" pod "a8c73d1b-a864-4534-944a-a543d04a2c3b" (UID: "a8c73d1b-a864-4534-944a-a543d04a2c3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.279995    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-workload-certs" pod "a8c73d1b-a864-4534-944a-a543d04a2c3b" (UID: "a8c73d1b-a864-4534-944a-a543d04a2c3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.280194    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-credential-socket" pod "75115e13-a71f-49ff-8bbc-6df7eebd4195" (UID: "75115e13-a71f-49ff-8bbc-6df7eebd4195"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.280210    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.280231    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.280245    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c73d1b-a864-4534-944a-a543d04a2c3b-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.280259    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.280272    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.280450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.280395    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-workload-socket" pod "a8c73d1b-a864-4534-944a-a543d04a2c3b" (UID: "a8c73d1b-a864-4534-944a-a543d04a2c3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.281854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.280595    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-workload-certs" pod "75115e13-a71f-49ff-8bbc-6df7eebd4195" (UID: "75115e13-a71f-49ff-8bbc-6df7eebd4195"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.281854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.280773    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-workload-socket" pod "75115e13-a71f-49ff-8bbc-6df7eebd4195" (UID: "75115e13-a71f-49ff-8bbc-6df7eebd4195"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.281854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.281058    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75115e13-a71f-49ff-8bbc-6df7eebd4195-istiod-ca-cert" pod "75115e13-a71f-49ff-8bbc-6df7eebd4195" (UID: "75115e13-a71f-49ff-8bbc-6df7eebd4195"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:15.282610 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.282585    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-envoy" pod "a8c73d1b-a864-4534-944a-a543d04a2c3b" (UID: "a8c73d1b-a864-4534-944a-a543d04a2c3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.283538 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.283516    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c73d1b-a864-4534-944a-a543d04a2c3b-kube-api-access-7kp6t" pod "a8c73d1b-a864-4534-944a-a543d04a2c3b" (UID: "a8c73d1b-a864-4534-944a-a543d04a2c3b"). InnerVolumeSpecName "kube-api-access-7kp6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:15.283655 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.283546    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-token" pod "a8c73d1b-a864-4534-944a-a543d04a2c3b" (UID: "a8c73d1b-a864-4534-944a-a543d04a2c3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:15.283655 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.283587    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-podinfo" pod "a8c73d1b-a864-4534-944a-a543d04a2c3b" (UID: "a8c73d1b-a864-4534-944a-a543d04a2c3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:13:15.283774 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.283686    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-envoy" pod "75115e13-a71f-49ff-8bbc-6df7eebd4195" (UID: "75115e13-a71f-49ff-8bbc-6df7eebd4195"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.283774 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.283698    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-podinfo" pod "75115e13-a71f-49ff-8bbc-6df7eebd4195" (UID: "75115e13-a71f-49ff-8bbc-6df7eebd4195"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:13:15.283912 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.283894    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-token" pod "75115e13-a71f-49ff-8bbc-6df7eebd4195" (UID: "75115e13-a71f-49ff-8bbc-6df7eebd4195"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:15.284592 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.284567    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75115e13-a71f-49ff-8bbc-6df7eebd4195-kube-api-access-2wp2h" pod "75115e13-a71f-49ff-8bbc-6df7eebd4195" (UID: "75115e13-a71f-49ff-8bbc-6df7eebd4195"). InnerVolumeSpecName "kube-api-access-2wp2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:15.366467 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.366384    5408 generic.go:356] "Generic (PLEG): container finished" podID="a8c73d1b-a864-4534-944a-a543d04a2c3b" containerID="476f128b0efd88bdb29619b29dc2a41c8ed9544f81491afd76a1430d130eee24" exitCode=0
Oct 30 20:13:15.366467 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.366435    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"
Oct 30 20:13:15.366707 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.366470    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84" event={"ID":"a8c73d1b-a864-4534-944a-a543d04a2c3b","Type":"ContainerDied","Data":"476f128b0efd88bdb29619b29dc2a41c8ed9544f81491afd76a1430d130eee24"}
Oct 30 20:13:15.366707 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.366529    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84" event={"ID":"a8c73d1b-a864-4534-944a-a543d04a2c3b","Type":"ContainerDied","Data":"112a9f7f1001bb160f7a61dc8d64341cd99bf49a6db32e9ae694e18b423e77d4"}
Oct 30 20:13:15.366707 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.366558    5408 scope.go:122] "RemoveContainer" containerID="476f128b0efd88bdb29619b29dc2a41c8ed9544f81491afd76a1430d130eee24"
Oct 30 20:13:15.368100 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.368073    5408 generic.go:356] "Generic (PLEG): container finished" podID="75115e13-a71f-49ff-8bbc-6df7eebd4195" containerID="6754b42b0f5681bf1388824942b0e8736f59e6217296b7acbc2c780254e55d07" exitCode=0
Oct 30 20:13:15.368255 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.368182    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"
Oct 30 20:13:15.368255 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.368195    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm" event={"ID":"75115e13-a71f-49ff-8bbc-6df7eebd4195","Type":"ContainerDied","Data":"6754b42b0f5681bf1388824942b0e8736f59e6217296b7acbc2c780254e55d07"}
Oct 30 20:13:15.368255 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.368244    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm" event={"ID":"75115e13-a71f-49ff-8bbc-6df7eebd4195","Type":"ContainerDied","Data":"9a94276aca74646ebc9af617bf3845ba0838ac1f44ad6187199da187b17b915c"}
Oct 30 20:13:15.377883 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.377854    5408 scope.go:122] "RemoveContainer" containerID="476f128b0efd88bdb29619b29dc2a41c8ed9544f81491afd76a1430d130eee24"
Oct 30 20:13:15.378299 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:13:15.378182    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476f128b0efd88bdb29619b29dc2a41c8ed9544f81491afd76a1430d130eee24\": container with ID starting with 476f128b0efd88bdb29619b29dc2a41c8ed9544f81491afd76a1430d130eee24 not found: ID does not exist" containerID="476f128b0efd88bdb29619b29dc2a41c8ed9544f81491afd76a1430d130eee24"
Oct 30 20:13:15.378299 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.378210    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476f128b0efd88bdb29619b29dc2a41c8ed9544f81491afd76a1430d130eee24"} err="failed to get container status \"476f128b0efd88bdb29619b29dc2a41c8ed9544f81491afd76a1430d130eee24\": rpc error: code = NotFound desc = could not find container \"476f128b0efd88bdb29619b29dc2a41c8ed9544f81491afd76a1430d130eee24\": container with ID starting with 476f128b0efd88bdb29619b29dc2a41c8ed9544f81491afd76a1430d130eee24 not found: ID does not exist"
Oct 30 20:13:15.378299 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.378226    5408 scope.go:122] "RemoveContainer" containerID="6754b42b0f5681bf1388824942b0e8736f59e6217296b7acbc2c780254e55d07"
Oct 30 20:13:15.381020 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.380996    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.381020 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.381025    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.381171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.381061    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.381171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.381074    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.381171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.381087    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kp6t\" (UniqueName: \"kubernetes.io/projected/a8c73d1b-a864-4534-944a-a543d04a2c3b-kube-api-access-7kp6t\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.381171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.381099    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.381171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.381115    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.381171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.381126    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c73d1b-a864-4534-944a-a543d04a2c3b-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.381171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.381135    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75115e13-a71f-49ff-8bbc-6df7eebd4195-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.381171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.381145    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75115e13-a71f-49ff-8bbc-6df7eebd4195-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.381171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.381153    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75115e13-a71f-49ff-8bbc-6df7eebd4195-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.381171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.381161    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wp2h\" (UniqueName: \"kubernetes.io/projected/75115e13-a71f-49ff-8bbc-6df7eebd4195-kube-api-access-2wp2h\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.381171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.381170    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c73d1b-a864-4534-944a-a543d04a2c3b-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.387190 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.387171    5408 scope.go:122] "RemoveContainer" containerID="6754b42b0f5681bf1388824942b0e8736f59e6217296b7acbc2c780254e55d07"
Oct 30 20:13:15.387478 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:13:15.387458    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6754b42b0f5681bf1388824942b0e8736f59e6217296b7acbc2c780254e55d07\": container with ID starting with 6754b42b0f5681bf1388824942b0e8736f59e6217296b7acbc2c780254e55d07 not found: ID does not exist" containerID="6754b42b0f5681bf1388824942b0e8736f59e6217296b7acbc2c780254e55d07"
Oct 30 20:13:15.387536 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.387484    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6754b42b0f5681bf1388824942b0e8736f59e6217296b7acbc2c780254e55d07"} err="failed to get container status \"6754b42b0f5681bf1388824942b0e8736f59e6217296b7acbc2c780254e55d07\": rpc error: code = NotFound desc = could not find container \"6754b42b0f5681bf1388824942b0e8736f59e6217296b7acbc2c780254e55d07\": container with ID starting with 6754b42b0f5681bf1388824942b0e8736f59e6217296b7acbc2c780254e55d07 not found: ID does not exist"
Oct 30 20:13:15.395426 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.395406    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"]
Oct 30 20:13:15.400580 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.400561    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-thb84"]
Oct 30 20:13:15.428540 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.428517    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"]
Oct 30 20:13:15.438824 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.438802    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-6htgm"]
Oct 30 20:13:15.542477 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.542445    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c2a0744-9e32-4896-914e-dfb9173923e7" path="/var/lib/kubelet/pods/6c2a0744-9e32-4896-914e-dfb9173923e7/volumes"
Oct 30 20:13:15.543376 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.543347    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75115e13-a71f-49ff-8bbc-6df7eebd4195" path="/var/lib/kubelet/pods/75115e13-a71f-49ff-8bbc-6df7eebd4195/volumes"
Oct 30 20:13:15.543936 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.543916    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c73d1b-a864-4534-944a-a543d04a2c3b" path="/var/lib/kubelet/pods/a8c73d1b-a864-4534-944a-a543d04a2c3b/volumes"
Oct 30 20:13:15.544981 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:15.544961    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" path="/var/lib/kubelet/pods/d7619cb9-6eef-4207-a4ee-a892f1920ff6/volumes"
Oct 30 20:13:16.379213 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.379069    5408 generic.go:356] "Generic (PLEG): container finished" podID="7577f068-6466-4add-8edf-691d113e8eb2" containerID="7301acfcb0c3035d3f5a59ff56d0a539d59054eb24a09e62efcc35ebc78912a8" exitCode=0
Oct 30 20:13:16.379213 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.379134    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp" event={"ID":"7577f068-6466-4add-8edf-691d113e8eb2","Type":"ContainerDied","Data":"7301acfcb0c3035d3f5a59ff56d0a539d59054eb24a09e62efcc35ebc78912a8"}
Oct 30 20:13:16.381298 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.381266    5408 generic.go:356] "Generic (PLEG): container finished" podID="bdae6413-f5be-452b-a5a7-d6ca4abb9081" containerID="9f27cdc03e3ec7fcc279e3f97fd84dde90b340224211a383ad72e68cb975209e" exitCode=0
Oct 30 20:13:16.381564 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.381337    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt" event={"ID":"bdae6413-f5be-452b-a5a7-d6ca4abb9081","Type":"ContainerDied","Data":"9f27cdc03e3ec7fcc279e3f97fd84dde90b340224211a383ad72e68cb975209e"}
Oct 30 20:13:16.407784 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.407732    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 20:13:16.424185 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.424165    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 20:13:16.592848 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.592814    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-workload-certs\") pod \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") "
Oct 30 20:13:16.593021 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.592858    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7577f068-6466-4add-8edf-691d113e8eb2-kube-api-access-gb9p5\" (UniqueName: \"kubernetes.io/projected/7577f068-6466-4add-8edf-691d113e8eb2-kube-api-access-gb9p5\") pod \"7577f068-6466-4add-8edf-691d113e8eb2\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") "
Oct 30 20:13:16.593021 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.592898    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-istio-data\") pod \"7577f068-6466-4add-8edf-691d113e8eb2\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") "
Oct 30 20:13:16.593021 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.592918    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-data\") pod \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") "
Oct 30 20:13:16.593021 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.592947    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-workload-socket\") pod \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") "
Oct 30 20:13:16.593021 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.592968    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-workload-socket\") pod \"7577f068-6466-4add-8edf-691d113e8eb2\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") "
Oct 30 20:13:16.593021 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593004    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istiod-ca-cert\") pod \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") "
Oct 30 20:13:16.593392 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593059    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-workload-certs\") pod \"7577f068-6466-4add-8edf-691d113e8eb2\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") "
Oct 30 20:13:16.593392 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593095    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7577f068-6466-4add-8edf-691d113e8eb2-istio-token\" (UniqueName: \"kubernetes.io/projected/7577f068-6466-4add-8edf-691d113e8eb2-istio-token\") pod \"7577f068-6466-4add-8edf-691d113e8eb2\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") "
Oct 30 20:13:16.593392 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593141    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-istio-data" pod "7577f068-6466-4add-8edf-691d113e8eb2" (UID: "7577f068-6466-4add-8edf-691d113e8eb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:16.593392 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593156    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-credential-socket\") pod \"7577f068-6466-4add-8edf-691d113e8eb2\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") "
Oct 30 20:13:16.593392 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593274    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7577f068-6466-4add-8edf-691d113e8eb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7577f068-6466-4add-8edf-691d113e8eb2-istiod-ca-cert\") pod \"7577f068-6466-4add-8edf-691d113e8eb2\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") "
Oct 30 20:13:16.593392 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593269    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-data" pod "bdae6413-f5be-452b-a5a7-d6ca4abb9081" (UID: "bdae6413-f5be-452b-a5a7-d6ca4abb9081"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:16.593392 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593323    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-workload-certs" pod "7577f068-6466-4add-8edf-691d113e8eb2" (UID: "7577f068-6466-4add-8edf-691d113e8eb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:16.593392 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593330    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-workload-socket" pod "bdae6413-f5be-452b-a5a7-d6ca4abb9081" (UID: "bdae6413-f5be-452b-a5a7-d6ca4abb9081"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:16.593392 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593297    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-workload-socket" pod "7577f068-6466-4add-8edf-691d113e8eb2" (UID: "7577f068-6466-4add-8edf-691d113e8eb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:16.593392 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593316    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-credential-socket\") pod \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") "
Oct 30 20:13:16.593904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593432    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-podinfo\") pod \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") "
Oct 30 20:13:16.593904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593492    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdae6413-f5be-452b-a5a7-d6ca4abb9081-kube-api-access-nn7k8\" (UniqueName: \"kubernetes.io/projected/bdae6413-f5be-452b-a5a7-d6ca4abb9081-kube-api-access-nn7k8\") pod \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") "
Oct 30 20:13:16.593904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593523    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-istio-envoy\") pod \"7577f068-6466-4add-8edf-691d113e8eb2\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") "
Oct 30 20:13:16.593904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593555    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7577f068-6466-4add-8edf-691d113e8eb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7577f068-6466-4add-8edf-691d113e8eb2-istio-podinfo\") pod \"7577f068-6466-4add-8edf-691d113e8eb2\" (UID: \"7577f068-6466-4add-8edf-691d113e8eb2\") "
Oct 30 20:13:16.593904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593564    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-credential-socket" pod "7577f068-6466-4add-8edf-691d113e8eb2" (UID: "7577f068-6466-4add-8edf-691d113e8eb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:16.593904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593582    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-workload-certs" pod "bdae6413-f5be-452b-a5a7-d6ca4abb9081" (UID: "bdae6413-f5be-452b-a5a7-d6ca4abb9081"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:16.593904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593590    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-token\" (UniqueName: \"kubernetes.io/projected/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-token\") pod \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") "
Oct 30 20:13:16.593904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593639    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-envoy\") pod \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\" (UID: \"bdae6413-f5be-452b-a5a7-d6ca4abb9081\") "
Oct 30 20:13:16.593904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593640    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istiod-ca-cert" pod "bdae6413-f5be-452b-a5a7-d6ca4abb9081" (UID: "bdae6413-f5be-452b-a5a7-d6ca4abb9081"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:16.593904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593699    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7577f068-6466-4add-8edf-691d113e8eb2-istiod-ca-cert" pod "7577f068-6466-4add-8edf-691d113e8eb2" (UID: "7577f068-6466-4add-8edf-691d113e8eb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:16.593904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.593839    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-credential-socket" pod "bdae6413-f5be-452b-a5a7-d6ca4abb9081" (UID: "bdae6413-f5be-452b-a5a7-d6ca4abb9081"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:16.594526 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.594113    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.594526 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.594136    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.594526 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.594149    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.594526 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.594162    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.594526 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.594175    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.594526 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.594188    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.594526 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.594204    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.594526 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.594217    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.594526 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.594230    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7577f068-6466-4add-8edf-691d113e8eb2-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.594526 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.594243    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.595921 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.595864    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7577f068-6466-4add-8edf-691d113e8eb2-kube-api-access-gb9p5" pod "7577f068-6466-4add-8edf-691d113e8eb2" (UID: "7577f068-6466-4add-8edf-691d113e8eb2"). InnerVolumeSpecName "kube-api-access-gb9p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:16.596117 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.596091    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7577f068-6466-4add-8edf-691d113e8eb2-istio-token" pod "7577f068-6466-4add-8edf-691d113e8eb2" (UID: "7577f068-6466-4add-8edf-691d113e8eb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:16.596235 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.596176    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdae6413-f5be-452b-a5a7-d6ca4abb9081-kube-api-access-nn7k8" pod "bdae6413-f5be-452b-a5a7-d6ca4abb9081" (UID: "bdae6413-f5be-452b-a5a7-d6ca4abb9081"). InnerVolumeSpecName "kube-api-access-nn7k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:16.596300 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.596252    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-token" pod "bdae6413-f5be-452b-a5a7-d6ca4abb9081" (UID: "bdae6413-f5be-452b-a5a7-d6ca4abb9081"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:16.596376 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.596356    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-istio-envoy" pod "7577f068-6466-4add-8edf-691d113e8eb2" (UID: "7577f068-6466-4add-8edf-691d113e8eb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:16.596415 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.596384    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-envoy" pod "bdae6413-f5be-452b-a5a7-d6ca4abb9081" (UID: "bdae6413-f5be-452b-a5a7-d6ca4abb9081"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:16.596837 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.596817    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7577f068-6466-4add-8edf-691d113e8eb2-istio-podinfo" pod "7577f068-6466-4add-8edf-691d113e8eb2" (UID: "7577f068-6466-4add-8edf-691d113e8eb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:13:16.597252 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.597236    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-podinfo" pod "bdae6413-f5be-452b-a5a7-d6ca4abb9081" (UID: "bdae6413-f5be-452b-a5a7-d6ca4abb9081"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:13:16.695391 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.695327    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7577f068-6466-4add-8edf-691d113e8eb2-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.695391 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.695352    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.695391 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.695363    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn7k8\" (UniqueName: \"kubernetes.io/projected/bdae6413-f5be-452b-a5a7-d6ca4abb9081-kube-api-access-nn7k8\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.695391 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.695373    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7577f068-6466-4add-8edf-691d113e8eb2-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.695391 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.695381    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7577f068-6466-4add-8edf-691d113e8eb2-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.695391 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.695389    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.695657 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.695398    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdae6413-f5be-452b-a5a7-d6ca4abb9081-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.695657 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:16.695406    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb9p5\" (UniqueName: \"kubernetes.io/projected/7577f068-6466-4add-8edf-691d113e8eb2-kube-api-access-gb9p5\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:17.390358 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:17.390328    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"
Oct 30 20:13:17.390808 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:17.390332    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp" event={"ID":"7577f068-6466-4add-8edf-691d113e8eb2","Type":"ContainerDied","Data":"7258dcc428a853143b6d29d711f2238629cc7c5ab149a6b6a9f1c5e93145b57c"}
Oct 30 20:13:17.390808 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:17.390489    5408 scope.go:122] "RemoveContainer" containerID="7301acfcb0c3035d3f5a59ff56d0a539d59054eb24a09e62efcc35ebc78912a8"
Oct 30 20:13:17.392105 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:17.392070    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"
Oct 30 20:13:17.392105 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:17.392074    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt" event={"ID":"bdae6413-f5be-452b-a5a7-d6ca4abb9081","Type":"ContainerDied","Data":"4c7849fb2f2b578817fd57c1812582dd55d06b95c0dd52cd9fe674c2754b6a8d"}
Oct 30 20:13:17.400688 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:17.400672    5408 scope.go:122] "RemoveContainer" containerID="9f27cdc03e3ec7fcc279e3f97fd84dde90b340224211a383ad72e68cb975209e"
Oct 30 20:13:17.430104 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:17.430082    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"]
Oct 30 20:13:17.445127 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:17.445103    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-s94pt"]
Oct 30 20:13:17.485108 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:17.485082    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"]
Oct 30 20:13:17.486593 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:17.486570    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-l7nkp"]
Oct 30 20:13:17.530743 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:17.530708    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7577f068-6466-4add-8edf-691d113e8eb2" path="/var/lib/kubelet/pods/7577f068-6466-4add-8edf-691d113e8eb2/volumes"
Oct 30 20:13:17.531282 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:17.531262    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdae6413-f5be-452b-a5a7-d6ca4abb9081" path="/var/lib/kubelet/pods/bdae6413-f5be-452b-a5a7-d6ca4abb9081/volumes"
Oct 30 20:13:21.719710 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:21.719677    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x"]
Oct 30 20:13:26.720993 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:26.720953    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x" podUID="c3e94e38-5327-48af-9f3d-a8541fe3ff0b" containerName="payload-pre-processing" containerID="cri-o://155e6ae005596d85269fcd4debcf04ccefd6e428906ec1b80d02c3a876862236" gracePeriod=25
Oct 30 20:13:26.967434 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:26.967412    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x"
Oct 30 20:13:27.020709 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:27.020677    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3e94e38-5327-48af-9f3d-a8541fe3ff0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e94e38-5327-48af-9f3d-a8541fe3ff0b-plugins-config-volume\") pod \"c3e94e38-5327-48af-9f3d-a8541fe3ff0b\" (UID: \"c3e94e38-5327-48af-9f3d-a8541fe3ff0b\") "
Oct 30 20:13:27.020875 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:27.020755    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e94e38-5327-48af-9f3d-a8541fe3ff0b-kube-api-access-4njfn\" (UniqueName: \"kubernetes.io/projected/c3e94e38-5327-48af-9f3d-a8541fe3ff0b-kube-api-access-4njfn\") pod \"c3e94e38-5327-48af-9f3d-a8541fe3ff0b\" (UID: \"c3e94e38-5327-48af-9f3d-a8541fe3ff0b\") "
Oct 30 20:13:27.021246 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:27.021215    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e94e38-5327-48af-9f3d-a8541fe3ff0b-plugins-config-volume" pod "c3e94e38-5327-48af-9f3d-a8541fe3ff0b" (UID: "c3e94e38-5327-48af-9f3d-a8541fe3ff0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:27.023315 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:27.023287    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e94e38-5327-48af-9f3d-a8541fe3ff0b-kube-api-access-4njfn" pod "c3e94e38-5327-48af-9f3d-a8541fe3ff0b" (UID: "c3e94e38-5327-48af-9f3d-a8541fe3ff0b"). InnerVolumeSpecName "kube-api-access-4njfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:27.123004 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:27.122959    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e94e38-5327-48af-9f3d-a8541fe3ff0b-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:27.123004 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:27.122997    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4njfn\" (UniqueName: \"kubernetes.io/projected/c3e94e38-5327-48af-9f3d-a8541fe3ff0b-kube-api-access-4njfn\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:27.449778 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:27.449691    5408 generic.go:356] "Generic (PLEG): container finished" podID="c3e94e38-5327-48af-9f3d-a8541fe3ff0b" containerID="155e6ae005596d85269fcd4debcf04ccefd6e428906ec1b80d02c3a876862236" exitCode=0
Oct 30 20:13:27.449778 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:27.449735    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x"
Oct 30 20:13:27.449986 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:27.449778    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x" event={"ID":"c3e94e38-5327-48af-9f3d-a8541fe3ff0b","Type":"ContainerDied","Data":"155e6ae005596d85269fcd4debcf04ccefd6e428906ec1b80d02c3a876862236"}
Oct 30 20:13:27.449986 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:27.449830    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x" event={"ID":"c3e94e38-5327-48af-9f3d-a8541fe3ff0b","Type":"ContainerDied","Data":"464b1604c1f43018a8f4f17a79ab98125c7048a2659848c2a616552d781710de"}
Oct 30 20:13:27.449986 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:27.449853    5408 scope.go:122] "RemoveContainer" containerID="155e6ae005596d85269fcd4debcf04ccefd6e428906ec1b80d02c3a876862236"
Oct 30 20:13:27.459529 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:27.459512    5408 scope.go:122] "RemoveContainer" containerID="155e6ae005596d85269fcd4debcf04ccefd6e428906ec1b80d02c3a876862236"
Oct 30 20:13:27.459798 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:13:27.459779    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"155e6ae005596d85269fcd4debcf04ccefd6e428906ec1b80d02c3a876862236\": container with ID starting with 155e6ae005596d85269fcd4debcf04ccefd6e428906ec1b80d02c3a876862236 not found: ID does not exist" containerID="155e6ae005596d85269fcd4debcf04ccefd6e428906ec1b80d02c3a876862236"
Oct 30 20:13:27.459863 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:27.459807    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"155e6ae005596d85269fcd4debcf04ccefd6e428906ec1b80d02c3a876862236"} err="failed to get container status \"155e6ae005596d85269fcd4debcf04ccefd6e428906ec1b80d02c3a876862236\": rpc error: code = NotFound desc = could not find container \"155e6ae005596d85269fcd4debcf04ccefd6e428906ec1b80d02c3a876862236\": container with ID starting with 155e6ae005596d85269fcd4debcf04ccefd6e428906ec1b80d02c3a876862236 not found: ID does not exist"
Oct 30 20:13:27.498062 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:27.498010    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x"]
Oct 30 20:13:27.510621 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:27.510587    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-3d0a78-94f9f84f9-ncv6x"]
Oct 30 20:13:27.530333 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:27.530308    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3e94e38-5327-48af-9f3d-a8541fe3ff0b" path="/var/lib/kubelet/pods/c3e94e38-5327-48af-9f3d-a8541fe3ff0b/volumes"
Oct 30 20:13:39.659646 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:13:39.659613    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:13:43.827874 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:43.827837    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"]
Oct 30 20:13:43.850152 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:43.850124    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"]
Oct 30 20:13:47.337649 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:47.337616    5408 scope.go:122] "RemoveContainer" containerID="736ccf33812f9fa040dfe650c49a85a2e917017a25af0537e65da7e1547bd8aa"
Oct 30 20:13:47.346930 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:47.346905    5408 scope.go:122] "RemoveContainer" containerID="0d717eb226e8b5c73e64bc593680dcd58c2638f38ebf3798f0ef13e4d07aab0a"
Oct 30 20:13:47.355570 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:47.355549    5408 scope.go:122] "RemoveContainer" containerID="5296966643fd29d112dcc525a53ef5a52c7e84bb6ed167fd2cad3d72808230c8"
Oct 30 20:13:48.985818 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:48.985779    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7"]
Oct 30 20:13:53.986725 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:53.986677    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7" podUID="268a8f7e-737f-4806-9f80-7d7de4494b00" containerName="payload-processing" containerID="cri-o://c97541349507ffb3384640e891e191106c5387c58b0946edba6fc2bc290f3ec6" gracePeriod=25
Oct 30 20:13:58.848248 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:58.848187    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main" containerID="cri-o://f028456ba5be02147eb8f1e3767153b3bd5bfb96d053c0cb7b43f4196280448d" gracePeriod=45
Oct 30 20:13:58.870215 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:58.870183    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main" containerID="cri-o://bb45b7251f3e1c7868218bd20f939a72f3b4ca5c7dfa48ef21ca065c71857ba3" gracePeriod=45
Oct 30 20:13:59.127895 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.127864    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 20:13:59.158136 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.158100    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 20:13:59.195318 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.195286    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-dshm\") pod \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") "
Oct 30 20:13:59.195560 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.195430    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-home\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-home\") pod \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") "
Oct 30 20:13:59.195560 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.195521    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f475ddeb-6235-4af9-a57c-7c4f5160fda9-kube-api-access-h9c5h\" (UniqueName: \"kubernetes.io/projected/f475ddeb-6235-4af9-a57c-7c4f5160fda9-kube-api-access-h9c5h\") pod \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") "
Oct 30 20:13:59.196171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.195567    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca13cd9-66b8-456e-8147-5a951cc47d88-kube-api-access-frqn8\" (UniqueName: \"kubernetes.io/projected/eca13cd9-66b8-456e-8147-5a951cc47d88-kube-api-access-frqn8\") pod \"eca13cd9-66b8-456e-8147-5a951cc47d88\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") "
Oct 30 20:13:59.196171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.195604    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-tmp-dir\") pod \"eca13cd9-66b8-456e-8147-5a951cc47d88\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") "
Oct 30 20:13:59.196171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.195605    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-home" pod "f475ddeb-6235-4af9-a57c-7c4f5160fda9" (UID: "f475ddeb-6235-4af9-a57c-7c4f5160fda9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:59.196171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.195638    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eca13cd9-66b8-456e-8147-5a951cc47d88-tls-certs\" (UniqueName: \"kubernetes.io/secret/eca13cd9-66b8-456e-8147-5a951cc47d88-tls-certs\") pod \"eca13cd9-66b8-456e-8147-5a951cc47d88\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") "
Oct 30 20:13:59.196171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.195679    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-model-cache\") pod \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") "
Oct 30 20:13:59.196171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.195710    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-home\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-home\") pod \"eca13cd9-66b8-456e-8147-5a951cc47d88\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") "
Oct 30 20:13:59.196171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.195747    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f475ddeb-6235-4af9-a57c-7c4f5160fda9-tls-certs\" (UniqueName: \"kubernetes.io/secret/f475ddeb-6235-4af9-a57c-7c4f5160fda9-tls-certs\") pod \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") "
Oct 30 20:13:59.196171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.195779    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-model-cache\") pod \"eca13cd9-66b8-456e-8147-5a951cc47d88\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") "
Oct 30 20:13:59.196171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.195873    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-dshm\") pod \"eca13cd9-66b8-456e-8147-5a951cc47d88\" (UID: \"eca13cd9-66b8-456e-8147-5a951cc47d88\") "
Oct 30 20:13:59.196171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.195952    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-tmp-dir\") pod \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\" (UID: \"f475ddeb-6235-4af9-a57c-7c4f5160fda9\") "
Oct 30 20:13:59.196171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.196075    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-model-cache" pod "f475ddeb-6235-4af9-a57c-7c4f5160fda9" (UID: "f475ddeb-6235-4af9-a57c-7c4f5160fda9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:59.197972 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.196318    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-tmp-dir" pod "f475ddeb-6235-4af9-a57c-7c4f5160fda9" (UID: "f475ddeb-6235-4af9-a57c-7c4f5160fda9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:59.197972 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.196575    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.197972 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.196908    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.197972 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.196927    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.197972 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.196582    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-home" pod "eca13cd9-66b8-456e-8147-5a951cc47d88" (UID: "eca13cd9-66b8-456e-8147-5a951cc47d88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:59.197972 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.197384    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-tmp-dir" pod "eca13cd9-66b8-456e-8147-5a951cc47d88" (UID: "eca13cd9-66b8-456e-8147-5a951cc47d88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:59.197972 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.197466    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-model-cache" pod "eca13cd9-66b8-456e-8147-5a951cc47d88" (UID: "eca13cd9-66b8-456e-8147-5a951cc47d88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:59.199125 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.199085    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca13cd9-66b8-456e-8147-5a951cc47d88-kube-api-access-frqn8" pod "eca13cd9-66b8-456e-8147-5a951cc47d88" (UID: "eca13cd9-66b8-456e-8147-5a951cc47d88"). InnerVolumeSpecName "kube-api-access-frqn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:59.199245 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.199174    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f475ddeb-6235-4af9-a57c-7c4f5160fda9-tls-certs" pod "f475ddeb-6235-4af9-a57c-7c4f5160fda9" (UID: "f475ddeb-6235-4af9-a57c-7c4f5160fda9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:59.199293 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.199269    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f475ddeb-6235-4af9-a57c-7c4f5160fda9-kube-api-access-h9c5h" pod "f475ddeb-6235-4af9-a57c-7c4f5160fda9" (UID: "f475ddeb-6235-4af9-a57c-7c4f5160fda9"). InnerVolumeSpecName "kube-api-access-h9c5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:59.201421 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.201385    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-dshm" pod "eca13cd9-66b8-456e-8147-5a951cc47d88" (UID: "eca13cd9-66b8-456e-8147-5a951cc47d88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:59.202704 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.202678    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-dshm" pod "f475ddeb-6235-4af9-a57c-7c4f5160fda9" (UID: "f475ddeb-6235-4af9-a57c-7c4f5160fda9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:59.202816 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.202775    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eca13cd9-66b8-456e-8147-5a951cc47d88-tls-certs" pod "eca13cd9-66b8-456e-8147-5a951cc47d88" (UID: "eca13cd9-66b8-456e-8147-5a951cc47d88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:59.298320 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.298284    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.298320 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.298316    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f475ddeb-6235-4af9-a57c-7c4f5160fda9-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.298320 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.298327    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9c5h\" (UniqueName: \"kubernetes.io/projected/f475ddeb-6235-4af9-a57c-7c4f5160fda9-kube-api-access-h9c5h\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.298562 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.298337    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frqn8\" (UniqueName: \"kubernetes.io/projected/eca13cd9-66b8-456e-8147-5a951cc47d88-kube-api-access-frqn8\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.298562 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.298346    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.298562 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.298354    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca13cd9-66b8-456e-8147-5a951cc47d88-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.298562 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.298362    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.298562 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.298370    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f475ddeb-6235-4af9-a57c-7c4f5160fda9-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.298562 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.298377    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca13cd9-66b8-456e-8147-5a951cc47d88-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.630903 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.630867    5408 generic.go:356] "Generic (PLEG): container finished" podID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerID="f028456ba5be02147eb8f1e3767153b3bd5bfb96d053c0cb7b43f4196280448d" exitCode=0
Oct 30 20:13:59.631082 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.630913    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"
Oct 30 20:13:59.631082 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.630955    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" event={"ID":"eca13cd9-66b8-456e-8147-5a951cc47d88","Type":"ContainerDied","Data":"f028456ba5be02147eb8f1e3767153b3bd5bfb96d053c0cb7b43f4196280448d"}
Oct 30 20:13:59.631082 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.630997    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs" event={"ID":"eca13cd9-66b8-456e-8147-5a951cc47d88","Type":"ContainerDied","Data":"309972040c11950ad992ba2f93216b082052ec853a32e37c69ae66f01a7c7c76"}
Oct 30 20:13:59.631082 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.631014    5408 scope.go:122] "RemoveContainer" containerID="f028456ba5be02147eb8f1e3767153b3bd5bfb96d053c0cb7b43f4196280448d"
Oct 30 20:13:59.632466 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.632432    5408 generic.go:356] "Generic (PLEG): container finished" podID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerID="bb45b7251f3e1c7868218bd20f939a72f3b4ca5c7dfa48ef21ca065c71857ba3" exitCode=0
Oct 30 20:13:59.632565 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.632478    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"
Oct 30 20:13:59.632565 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.632550    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" event={"ID":"f475ddeb-6235-4af9-a57c-7c4f5160fda9","Type":"ContainerDied","Data":"bb45b7251f3e1c7868218bd20f939a72f3b4ca5c7dfa48ef21ca065c71857ba3"}
Oct 30 20:13:59.632686 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.632580    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh" event={"ID":"f475ddeb-6235-4af9-a57c-7c4f5160fda9","Type":"ContainerDied","Data":"1bf2a83d4778861559b9207c9c151c4c035322f46db58a44410139300e15f0c2"}
Oct 30 20:13:59.642898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.642876    5408 scope.go:122] "RemoveContainer" containerID="f028456ba5be02147eb8f1e3767153b3bd5bfb96d053c0cb7b43f4196280448d"
Oct 30 20:13:59.643601 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:13:59.643572    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f028456ba5be02147eb8f1e3767153b3bd5bfb96d053c0cb7b43f4196280448d\": container with ID starting with f028456ba5be02147eb8f1e3767153b3bd5bfb96d053c0cb7b43f4196280448d not found: ID does not exist" containerID="f028456ba5be02147eb8f1e3767153b3bd5bfb96d053c0cb7b43f4196280448d"
Oct 30 20:13:59.643721 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.643609    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f028456ba5be02147eb8f1e3767153b3bd5bfb96d053c0cb7b43f4196280448d"} err="failed to get container status \"f028456ba5be02147eb8f1e3767153b3bd5bfb96d053c0cb7b43f4196280448d\": rpc error: code = NotFound desc = could not find container \"f028456ba5be02147eb8f1e3767153b3bd5bfb96d053c0cb7b43f4196280448d\": container with ID starting with f028456ba5be02147eb8f1e3767153b3bd5bfb96d053c0cb7b43f4196280448d not found: ID does not exist"
Oct 30 20:13:59.643721 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.643631    5408 scope.go:122] "RemoveContainer" containerID="bb45b7251f3e1c7868218bd20f939a72f3b4ca5c7dfa48ef21ca065c71857ba3"
Oct 30 20:13:59.654378 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.654358    5408 scope.go:122] "RemoveContainer" containerID="bb45b7251f3e1c7868218bd20f939a72f3b4ca5c7dfa48ef21ca065c71857ba3"
Oct 30 20:13:59.654455 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.654436    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"]
Oct 30 20:13:59.654821 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:13:59.654786    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb45b7251f3e1c7868218bd20f939a72f3b4ca5c7dfa48ef21ca065c71857ba3\": container with ID starting with bb45b7251f3e1c7868218bd20f939a72f3b4ca5c7dfa48ef21ca065c71857ba3 not found: ID does not exist" containerID="bb45b7251f3e1c7868218bd20f939a72f3b4ca5c7dfa48ef21ca065c71857ba3"
Oct 30 20:13:59.655006 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.654973    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb45b7251f3e1c7868218bd20f939a72f3b4ca5c7dfa48ef21ca065c71857ba3"} err="failed to get container status \"bb45b7251f3e1c7868218bd20f939a72f3b4ca5c7dfa48ef21ca065c71857ba3\": rpc error: code = NotFound desc = could not find container \"bb45b7251f3e1c7868218bd20f939a72f3b4ca5c7dfa48ef21ca065c71857ba3\": container with ID starting with bb45b7251f3e1c7868218bd20f939a72f3b4ca5c7dfa48ef21ca065c71857ba3 not found: ID does not exist"
Oct 30 20:13:59.656159 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.656135    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/distinct-w0-ced832-kserve-7f9c4956d-25ncs"]
Oct 30 20:13:59.667496 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.667466    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"]
Oct 30 20:13:59.672655 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:13:59.672630    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/simulator-w0-ced832-kserve-86c4d4968d-9tfqh"]
Oct 30 20:14:01.077111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:01.077077    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7" podUID="268a8f7e-737f-4806-9f80-7d7de4494b00" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:14:01.530332 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:01.530301    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" path="/var/lib/kubelet/pods/eca13cd9-66b8-456e-8147-5a951cc47d88/volumes"
Oct 30 20:14:01.530729 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:01.530715    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" path="/var/lib/kubelet/pods/f475ddeb-6235-4af9-a57c-7c4f5160fda9/volumes"
Oct 30 20:14:09.659230 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:14:09.659193    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:14:09.682046 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:09.682006    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"]
Oct 30 20:14:09.682294 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:09.682268    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd" podUID="66fb1cac-4e6d-4d5a-812d-5054e1676769" containerName="maas-api" containerID="cri-o://b185cd94638e9903bebc2f25f6c7f8bc5d92c3a163b1f399bbef3844a9aa88c9" gracePeriod=30
Oct 30 20:14:09.994870 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:09.994844    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"
Oct 30 20:14:10.127087 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.127020    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66fb1cac-4e6d-4d5a-812d-5054e1676769-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66fb1cac-4e6d-4d5a-812d-5054e1676769-openshift-service-ca\") pod \"66fb1cac-4e6d-4d5a-812d-5054e1676769\" (UID: \"66fb1cac-4e6d-4d5a-812d-5054e1676769\") "
Oct 30 20:14:10.127285 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.127163    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66fb1cac-4e6d-4d5a-812d-5054e1676769-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66fb1cac-4e6d-4d5a-812d-5054e1676769-metrics-certs\") pod \"66fb1cac-4e6d-4d5a-812d-5054e1676769\" (UID: \"66fb1cac-4e6d-4d5a-812d-5054e1676769\") "
Oct 30 20:14:10.127285 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.127189    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66fb1cac-4e6d-4d5a-812d-5054e1676769-kube-api-access-nj8pb\" (UniqueName: \"kubernetes.io/projected/66fb1cac-4e6d-4d5a-812d-5054e1676769-kube-api-access-nj8pb\") pod \"66fb1cac-4e6d-4d5a-812d-5054e1676769\" (UID: \"66fb1cac-4e6d-4d5a-812d-5054e1676769\") "
Oct 30 20:14:10.127285 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.127229    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66fb1cac-4e6d-4d5a-812d-5054e1676769-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66fb1cac-4e6d-4d5a-812d-5054e1676769-maas-api-tls\") pod \"66fb1cac-4e6d-4d5a-812d-5054e1676769\" (UID: \"66fb1cac-4e6d-4d5a-812d-5054e1676769\") "
Oct 30 20:14:10.127470 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.127425    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fb1cac-4e6d-4d5a-812d-5054e1676769-openshift-service-ca" pod "66fb1cac-4e6d-4d5a-812d-5054e1676769" (UID: "66fb1cac-4e6d-4d5a-812d-5054e1676769"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:10.127803 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.127633    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66fb1cac-4e6d-4d5a-812d-5054e1676769-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.130178 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.130108    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66fb1cac-4e6d-4d5a-812d-5054e1676769-metrics-certs" pod "66fb1cac-4e6d-4d5a-812d-5054e1676769" (UID: "66fb1cac-4e6d-4d5a-812d-5054e1676769"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:14:10.131448 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.131421    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66fb1cac-4e6d-4d5a-812d-5054e1676769-maas-api-tls" pod "66fb1cac-4e6d-4d5a-812d-5054e1676769" (UID: "66fb1cac-4e6d-4d5a-812d-5054e1676769"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:14:10.148006 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.147652    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fb1cac-4e6d-4d5a-812d-5054e1676769-kube-api-access-nj8pb" pod "66fb1cac-4e6d-4d5a-812d-5054e1676769" (UID: "66fb1cac-4e6d-4d5a-812d-5054e1676769"). InnerVolumeSpecName "kube-api-access-nj8pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:10.228354 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.228282    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66fb1cac-4e6d-4d5a-812d-5054e1676769-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.228354 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.228308    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66fb1cac-4e6d-4d5a-812d-5054e1676769-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.228354 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.228320    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj8pb\" (UniqueName: \"kubernetes.io/projected/66fb1cac-4e6d-4d5a-812d-5054e1676769-kube-api-access-nj8pb\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:10.690595 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.690558    5408 generic.go:356] "Generic (PLEG): container finished" podID="66fb1cac-4e6d-4d5a-812d-5054e1676769" containerID="b185cd94638e9903bebc2f25f6c7f8bc5d92c3a163b1f399bbef3844a9aa88c9" exitCode=0
Oct 30 20:14:10.691140 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.690610    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"
Oct 30 20:14:10.691140 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.690643    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd" event={"ID":"66fb1cac-4e6d-4d5a-812d-5054e1676769","Type":"ContainerDied","Data":"b185cd94638e9903bebc2f25f6c7f8bc5d92c3a163b1f399bbef3844a9aa88c9"}
Oct 30 20:14:10.691140 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.690700    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd" event={"ID":"66fb1cac-4e6d-4d5a-812d-5054e1676769","Type":"ContainerDied","Data":"639d2e26d7c2a2f8e08ef6393c5e1db11efd0e892de54f53afa801a14389e1c9"}
Oct 30 20:14:10.691140 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.690721    5408 scope.go:122] "RemoveContainer" containerID="b185cd94638e9903bebc2f25f6c7f8bc5d92c3a163b1f399bbef3844a9aa88c9"
Oct 30 20:14:10.700323 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.700306    5408 scope.go:122] "RemoveContainer" containerID="b185cd94638e9903bebc2f25f6c7f8bc5d92c3a163b1f399bbef3844a9aa88c9"
Oct 30 20:14:10.700582 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:14:10.700561    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b185cd94638e9903bebc2f25f6c7f8bc5d92c3a163b1f399bbef3844a9aa88c9\": container with ID starting with b185cd94638e9903bebc2f25f6c7f8bc5d92c3a163b1f399bbef3844a9aa88c9 not found: ID does not exist" containerID="b185cd94638e9903bebc2f25f6c7f8bc5d92c3a163b1f399bbef3844a9aa88c9"
Oct 30 20:14:10.700639 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.700591    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b185cd94638e9903bebc2f25f6c7f8bc5d92c3a163b1f399bbef3844a9aa88c9"} err="failed to get container status \"b185cd94638e9903bebc2f25f6c7f8bc5d92c3a163b1f399bbef3844a9aa88c9\": rpc error: code = NotFound desc = could not find container \"b185cd94638e9903bebc2f25f6c7f8bc5d92c3a163b1f399bbef3844a9aa88c9\": container with ID starting with b185cd94638e9903bebc2f25f6c7f8bc5d92c3a163b1f399bbef3844a9aa88c9 not found: ID does not exist"
Oct 30 20:14:10.715412 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.715391    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"]
Oct 30 20:14:10.721118 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:10.721098    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ced832-79c6f5c54d-wzgcd"]
Oct 30 20:14:11.077631 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:11.077598    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7" podUID="268a8f7e-737f-4806-9f80-7d7de4494b00" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:14:11.530346 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:11.530313    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66fb1cac-4e6d-4d5a-812d-5054e1676769" path="/var/lib/kubelet/pods/66fb1cac-4e6d-4d5a-812d-5054e1676769/volumes"
Oct 30 20:14:14.250117 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:14.250060    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7"
Oct 30 20:14:14.379169 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:14.379132    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/268a8f7e-737f-4806-9f80-7d7de4494b00-kube-api-access-mzknb\" (UniqueName: \"kubernetes.io/projected/268a8f7e-737f-4806-9f80-7d7de4494b00-kube-api-access-mzknb\") pod \"268a8f7e-737f-4806-9f80-7d7de4494b00\" (UID: \"268a8f7e-737f-4806-9f80-7d7de4494b00\") "
Oct 30 20:14:14.379362 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:14.379307    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/268a8f7e-737f-4806-9f80-7d7de4494b00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268a8f7e-737f-4806-9f80-7d7de4494b00-plugins-config-volume\") pod \"268a8f7e-737f-4806-9f80-7d7de4494b00\" (UID: \"268a8f7e-737f-4806-9f80-7d7de4494b00\") "
Oct 30 20:14:14.379653 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:14.379629    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268a8f7e-737f-4806-9f80-7d7de4494b00-plugins-config-volume" pod "268a8f7e-737f-4806-9f80-7d7de4494b00" (UID: "268a8f7e-737f-4806-9f80-7d7de4494b00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:14.381254 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:14.381232    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268a8f7e-737f-4806-9f80-7d7de4494b00-kube-api-access-mzknb" pod "268a8f7e-737f-4806-9f80-7d7de4494b00" (UID: "268a8f7e-737f-4806-9f80-7d7de4494b00"). InnerVolumeSpecName "kube-api-access-mzknb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:14.480486 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:14.480460    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268a8f7e-737f-4806-9f80-7d7de4494b00-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:14.480486 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:14.480485    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzknb\" (UniqueName: \"kubernetes.io/projected/268a8f7e-737f-4806-9f80-7d7de4494b00-kube-api-access-mzknb\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:14.712984 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:14.712891    5408 generic.go:356] "Generic (PLEG): container finished" podID="268a8f7e-737f-4806-9f80-7d7de4494b00" containerID="c97541349507ffb3384640e891e191106c5387c58b0946edba6fc2bc290f3ec6" exitCode=0
Oct 30 20:14:14.712984 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:14.712935    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7"
Oct 30 20:14:14.713247 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:14.712975    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7" event={"ID":"268a8f7e-737f-4806-9f80-7d7de4494b00","Type":"ContainerDied","Data":"c97541349507ffb3384640e891e191106c5387c58b0946edba6fc2bc290f3ec6"}
Oct 30 20:14:14.713247 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:14.713023    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7" event={"ID":"268a8f7e-737f-4806-9f80-7d7de4494b00","Type":"ContainerDied","Data":"071ffee22aa020b5387416afab821fe4dc300e1747285149e7e6c9410471ffbe"}
Oct 30 20:14:14.713247 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:14.713059    5408 scope.go:122] "RemoveContainer" containerID="c97541349507ffb3384640e891e191106c5387c58b0946edba6fc2bc290f3ec6"
Oct 30 20:14:14.721739 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:14.721708    5408 scope.go:122] "RemoveContainer" containerID="c97541349507ffb3384640e891e191106c5387c58b0946edba6fc2bc290f3ec6"
Oct 30 20:14:14.722019 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:14:14.722003    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c97541349507ffb3384640e891e191106c5387c58b0946edba6fc2bc290f3ec6\": container with ID starting with c97541349507ffb3384640e891e191106c5387c58b0946edba6fc2bc290f3ec6 not found: ID does not exist" containerID="c97541349507ffb3384640e891e191106c5387c58b0946edba6fc2bc290f3ec6"
Oct 30 20:14:14.722085 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:14.722056    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97541349507ffb3384640e891e191106c5387c58b0946edba6fc2bc290f3ec6"} err="failed to get container status \"c97541349507ffb3384640e891e191106c5387c58b0946edba6fc2bc290f3ec6\": rpc error: code = NotFound desc = could not find container \"c97541349507ffb3384640e891e191106c5387c58b0946edba6fc2bc290f3ec6\": container with ID starting with c97541349507ffb3384640e891e191106c5387c58b0946edba6fc2bc290f3ec6 not found: ID does not exist"
Oct 30 20:14:14.737105 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:14.737085    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7"]
Oct 30 20:14:14.741677 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:14.741655    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ced832-6995fbff6-rfcl7"]
Oct 30 20:14:15.532359 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:15.532320    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="268a8f7e-737f-4806-9f80-7d7de4494b00" path="/var/lib/kubelet/pods/268a8f7e-737f-4806-9f80-7d7de4494b00/volumes"
Oct 30 20:14:17.158784 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.158754    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"]
Oct 30 20:14:17.159216 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159202    5408 cpu_manager.go:414] "containerMap: removing container" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main"
Oct 30 20:14:17.159262 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159216    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main"
Oct 30 20:14:17.159262 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159232    5408 cpu_manager.go:414] "containerMap: removing container" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy"
Oct 30 20:14:17.159262 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159237    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy"
Oct 30 20:14:17.159262 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159244    5408 cpu_manager.go:414] "containerMap: removing container" podUID="aecc895e-a6d4-4fc6-8e55-92b83f17fcde" containerName="payload-processing"
Oct 30 20:14:17.159262 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159249    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aecc895e-a6d4-4fc6-8e55-92b83f17fcde" containerName="payload-processing"
Oct 30 20:14:17.159262 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159258    5408 cpu_manager.go:414] "containerMap: removing container" podUID="d3cbd0cd-db30-4181-9271-fab50895d832" containerName="main"
Oct 30 20:14:17.159262 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159262    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3cbd0cd-db30-4181-9271-fab50895d832" containerName="main"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159272    5408 cpu_manager.go:414] "containerMap: removing container" podUID="50041f27-99a2-47ba-944a-9693459b8c80" containerName="istio-proxy"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159277    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50041f27-99a2-47ba-944a-9693459b8c80" containerName="istio-proxy"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159285    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159290    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159297    5408 cpu_manager.go:414] "containerMap: removing container" podUID="7577f068-6466-4add-8edf-691d113e8eb2" containerName="istio-proxy"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159301    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7577f068-6466-4add-8edf-691d113e8eb2" containerName="istio-proxy"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159307    5408 cpu_manager.go:414] "containerMap: removing container" podUID="3558d905-d0a3-4b3f-8c5c-dac3353f4b1c" containerName="main"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159311    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3558d905-d0a3-4b3f-8c5c-dac3353f4b1c" containerName="main"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159318    5408 cpu_manager.go:414] "containerMap: removing container" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159322    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159327    5408 cpu_manager.go:414] "containerMap: removing container" podUID="bdae6413-f5be-452b-a5a7-d6ca4abb9081" containerName="istio-proxy"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159332    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdae6413-f5be-452b-a5a7-d6ca4abb9081" containerName="istio-proxy"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159339    5408 cpu_manager.go:414] "containerMap: removing container" podUID="75115e13-a71f-49ff-8bbc-6df7eebd4195" containerName="istio-proxy"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159343    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75115e13-a71f-49ff-8bbc-6df7eebd4195" containerName="istio-proxy"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159349    5408 cpu_manager.go:414] "containerMap: removing container" podUID="c3e94e38-5327-48af-9f3d-a8541fe3ff0b" containerName="payload-pre-processing"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159354    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e94e38-5327-48af-9f3d-a8541fe3ff0b" containerName="payload-pre-processing"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159361    5408 cpu_manager.go:414] "containerMap: removing container" podUID="d8c8ac1e-8cf9-4a86-affa-9a2745709749" containerName="maas-api"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159366    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c8ac1e-8cf9-4a86-affa-9a2745709749" containerName="maas-api"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159372    5408 cpu_manager.go:414] "containerMap: removing container" podUID="6c2a0744-9e32-4896-914e-dfb9173923e7" containerName="istio-proxy"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159377    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c2a0744-9e32-4896-914e-dfb9173923e7" containerName="istio-proxy"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159383    5408 cpu_manager.go:414] "containerMap: removing container" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159387    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159394    5408 cpu_manager.go:414] "containerMap: removing container" podUID="e0c23612-28be-4e5f-aa95-71c40e82450e" containerName="istio-proxy"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159398    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c23612-28be-4e5f-aa95-71c40e82450e" containerName="istio-proxy"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159403    5408 cpu_manager.go:414] "containerMap: removing container" podUID="4ddd1055-47a1-4c05-bd03-ff3c493f32a7" containerName="istio-proxy"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159408    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ddd1055-47a1-4c05-bd03-ff3c493f32a7" containerName="istio-proxy"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159414    5408 cpu_manager.go:414] "containerMap: removing container" podUID="66fb1cac-4e6d-4d5a-812d-5054e1676769" containerName="maas-api"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159418    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66fb1cac-4e6d-4d5a-812d-5054e1676769" containerName="maas-api"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159424    5408 cpu_manager.go:414] "containerMap: removing container" podUID="415f31e1-c808-4fb0-bb8c-33f928248892" containerName="payload-processing"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159429    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415f31e1-c808-4fb0-bb8c-33f928248892" containerName="payload-processing"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159435    5408 cpu_manager.go:414] "containerMap: removing container" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159440    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159446    5408 cpu_manager.go:414] "containerMap: removing container" podUID="a8c73d1b-a864-4534-944a-a543d04a2c3b" containerName="istio-proxy"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159450    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c73d1b-a864-4534-944a-a543d04a2c3b" containerName="istio-proxy"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159458    5408 cpu_manager.go:414] "containerMap: removing container" podUID="651ba59f-3f1f-479d-805e-30a17661c602" containerName="payload-pre-processing"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159463    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651ba59f-3f1f-479d-805e-30a17661c602" containerName="payload-pre-processing"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159470    5408 cpu_manager.go:414] "containerMap: removing container" podUID="a848a44a-9e0a-47aa-811f-5a9930b9bd9f" containerName="payload-processing"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159474    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a848a44a-9e0a-47aa-811f-5a9930b9bd9f" containerName="payload-processing"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159482    5408 cpu_manager.go:414] "containerMap: removing container" podUID="39539848-97ad-40d8-8082-d27a15143388" containerName="payload-processing"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159486    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39539848-97ad-40d8-8082-d27a15143388" containerName="payload-processing"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159493    5408 cpu_manager.go:414] "containerMap: removing container" podUID="90d486b1-ecce-40a9-8353-04127a5111df" containerName="main"
Oct 30 20:14:17.159481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159497    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90d486b1-ecce-40a9-8353-04127a5111df" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159504    5408 cpu_manager.go:414] "containerMap: removing container" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159509    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159517    5408 cpu_manager.go:414] "containerMap: removing container" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159521    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159528    5408 cpu_manager.go:414] "containerMap: removing container" podUID="38fa73eb-88a7-4868-8e33-a407edc23aef" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159533    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38fa73eb-88a7-4868-8e33-a407edc23aef" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159540    5408 cpu_manager.go:414] "containerMap: removing container" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159544    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159550    5408 cpu_manager.go:414] "containerMap: removing container" podUID="88493f4b-7ec6-4f72-92df-03fb99d9c612" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159554    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88493f4b-7ec6-4f72-92df-03fb99d9c612" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159560    5408 cpu_manager.go:414] "containerMap: removing container" podUID="7c77d77b-fee8-48c4-a717-5c17f9f02453" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159564    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c77d77b-fee8-48c4-a717-5c17f9f02453" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159571    5408 cpu_manager.go:414] "containerMap: removing container" podUID="25efa968-ddb4-4893-8209-176519fb0224" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159575    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25efa968-ddb4-4893-8209-176519fb0224" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159580    5408 cpu_manager.go:414] "containerMap: removing container" podUID="b19efa61-42ee-40ce-98b4-6b57dd46b3c1" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159584    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b19efa61-42ee-40ce-98b4-6b57dd46b3c1" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159592    5408 cpu_manager.go:414] "containerMap: removing container" podUID="c024192c-6410-4363-89b9-4b0e4fd5e4dc" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159596    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c024192c-6410-4363-89b9-4b0e4fd5e4dc" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159601    5408 cpu_manager.go:414] "containerMap: removing container" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159606    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159610    5408 cpu_manager.go:414] "containerMap: removing container" podUID="c7bf362b-a122-4417-813a-1c1f30c05297" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159615    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7bf362b-a122-4417-813a-1c1f30c05297" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159622    5408 cpu_manager.go:414] "containerMap: removing container" podUID="e0c23612-28be-4e5f-aa95-71c40e82450e" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159626    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c23612-28be-4e5f-aa95-71c40e82450e" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159632    5408 cpu_manager.go:414] "containerMap: removing container" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159636    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159642    5408 cpu_manager.go:414] "containerMap: removing container" podUID="686b83f0-b8a4-4210-a66d-e92c30e7fc8c" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159646    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="686b83f0-b8a4-4210-a66d-e92c30e7fc8c" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159652    5408 cpu_manager.go:414] "containerMap: removing container" podUID="e48c3cc5-30d3-4bc8-98a7-548d78595ac6" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159657    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e48c3cc5-30d3-4bc8-98a7-548d78595ac6" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159664    5408 cpu_manager.go:414] "containerMap: removing container" podUID="3fc2d2e5-3aab-4f01-997b-07aa7841dd73" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159668    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc2d2e5-3aab-4f01-997b-07aa7841dd73" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159674    5408 cpu_manager.go:414] "containerMap: removing container" podUID="268a8f7e-737f-4806-9f80-7d7de4494b00" containerName="payload-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159679    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="268a8f7e-737f-4806-9f80-7d7de4494b00" containerName="payload-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159685    5408 cpu_manager.go:414] "containerMap: removing container" podUID="b599d3f2-4e57-4072-b8f9-02e0d8422678" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159690    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b599d3f2-4e57-4072-b8f9-02e0d8422678" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159697    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159702    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159709    5408 cpu_manager.go:414] "containerMap: removing container" podUID="688f0cb2-8d4a-47ab-ab79-fb1a54109e48" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159713    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="688f0cb2-8d4a-47ab-ab79-fb1a54109e48" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159720    5408 cpu_manager.go:414] "containerMap: removing container" podUID="50041f27-99a2-47ba-944a-9693459b8c80" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159726    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50041f27-99a2-47ba-944a-9693459b8c80" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159734    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f99c11d2-540e-434a-8271-92ea41a6b437" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159739    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99c11d2-540e-434a-8271-92ea41a6b437" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159744    5408 cpu_manager.go:414] "containerMap: removing container" podUID="9b56dd57-5a11-4834-a128-5aa9e2385157" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159749    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b56dd57-5a11-4834-a128-5aa9e2385157" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159754    5408 cpu_manager.go:414] "containerMap: removing container" podUID="c27eb3c2-6159-459c-8ef1-6ef50c9b62cc" containerName="payload-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159758    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27eb3c2-6159-459c-8ef1-6ef50c9b62cc" containerName="payload-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159766    5408 cpu_manager.go:414] "containerMap: removing container" podUID="d7a32800-c4b4-46eb-abcd-fee2c0d88a29" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159770    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a32800-c4b4-46eb-abcd-fee2c0d88a29" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159776    5408 cpu_manager.go:414] "containerMap: removing container" podUID="723bb78d-b254-4cea-bf5e-6e5609a8a699" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159781    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="723bb78d-b254-4cea-bf5e-6e5609a8a699" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159786    5408 cpu_manager.go:414] "containerMap: removing container" podUID="e8795c70-cfa2-46d3-85a9-3af34e81144d" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159790    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8795c70-cfa2-46d3-85a9-3af34e81144d" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159796    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f99c11d2-540e-434a-8271-92ea41a6b437" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159801    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99c11d2-540e-434a-8271-92ea41a6b437" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159807    5408 cpu_manager.go:414] "containerMap: removing container" podUID="5b518877-72bf-4743-8f2e-daa5f9800392" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159811    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b518877-72bf-4743-8f2e-daa5f9800392" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159817    5408 cpu_manager.go:414] "containerMap: removing container" podUID="c0bb64d8-6abf-45cc-8feb-d7ba98073055" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159821    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0bb64d8-6abf-45cc-8feb-d7ba98073055" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159826    5408 cpu_manager.go:414] "containerMap: removing container" podUID="7b091309-1316-445d-bfee-4ba4ba8b8319" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159830    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b091309-1316-445d-bfee-4ba4ba8b8319" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159837    5408 cpu_manager.go:414] "containerMap: removing container" podUID="195cd4cc-15b4-470c-b139-f0dd93530204" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159841    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="195cd4cc-15b4-470c-b139-f0dd93530204" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159848    5408 cpu_manager.go:414] "containerMap: removing container" podUID="eb4fd624-deaf-4da1-9183-e83b29f52d64" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159853    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb4fd624-deaf-4da1-9183-e83b29f52d64" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159858    5408 cpu_manager.go:414] "containerMap: removing container" podUID="cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1" containerName="payload-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159862    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1" containerName="payload-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159925    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="910fde4a-f4d1-4547-a060-2cf6888e5942" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159934    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="88493f4b-7ec6-4f72-92df-03fb99d9c612" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159942    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c77d77b-fee8-48c4-a717-5c17f9f02453" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159949    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="25efa968-ddb4-4893-8209-176519fb0224" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159954    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="b19efa61-42ee-40ce-98b4-6b57dd46b3c1" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159962    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="c024192c-6410-4363-89b9-4b0e4fd5e4dc" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159968    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e5be67-6a55-4e8f-94dc-1a64e11d7f65" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159973    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7bf362b-a122-4417-813a-1c1f30c05297" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159979    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c23612-28be-4e5f-aa95-71c40e82450e" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159984    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c65275-2119-45ac-8393-c29bce507f33" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159990    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="686b83f0-b8a4-4210-a66d-e92c30e7fc8c" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.159997    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="e48c3cc5-30d3-4bc8-98a7-548d78595ac6" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160002    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc2d2e5-3aab-4f01-997b-07aa7841dd73" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160009    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="268a8f7e-737f-4806-9f80-7d7de4494b00" containerName="payload-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160014    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="b599d3f2-4e57-4072-b8f9-02e0d8422678" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160020    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="f475ddeb-6235-4af9-a57c-7c4f5160fda9" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160025    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="688f0cb2-8d4a-47ab-ab79-fb1a54109e48" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160056    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="50041f27-99a2-47ba-944a-9693459b8c80" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160061    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99c11d2-540e-434a-8271-92ea41a6b437" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160066    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b56dd57-5a11-4834-a128-5aa9e2385157" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160071    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27eb3c2-6159-459c-8ef1-6ef50c9b62cc" containerName="payload-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160076    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a32800-c4b4-46eb-abcd-fee2c0d88a29" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160082    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="723bb78d-b254-4cea-bf5e-6e5609a8a699" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160086    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8795c70-cfa2-46d3-85a9-3af34e81144d" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160091    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99c11d2-540e-434a-8271-92ea41a6b437" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160096    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0bb64d8-6abf-45cc-8feb-d7ba98073055" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160103    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b518877-72bf-4743-8f2e-daa5f9800392" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160107    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b091309-1316-445d-bfee-4ba4ba8b8319" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160115    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="195cd4cc-15b4-470c-b139-f0dd93530204" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160121    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb4fd624-deaf-4da1-9183-e83b29f52d64" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160125    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb1e3f93-8a9d-4ea9-8443-d15fb788e9b1" containerName="payload-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160131    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="920fc884-8d5e-40e5-9e67-550521b12e3e" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160136    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7619cb9-6eef-4207-a4ee-a892f1920ff6" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160140    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="aecc895e-a6d4-4fc6-8e55-92b83f17fcde" containerName="payload-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160146    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3cbd0cd-db30-4181-9271-fab50895d832" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160153    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="f141028a-4c85-40ca-b0aa-cdc0838170d1" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160162    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="7577f068-6466-4add-8edf-691d113e8eb2" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160170    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="3558d905-d0a3-4b3f-8c5c-dac3353f4b1c" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160178    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a886ba9-fe5e-4739-bb26-7bdb235e6e64" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160185    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdae6413-f5be-452b-a5a7-d6ca4abb9081" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160194    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="75115e13-a71f-49ff-8bbc-6df7eebd4195" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160199    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e94e38-5327-48af-9f3d-a8541fe3ff0b" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160204    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c8ac1e-8cf9-4a86-affa-9a2745709749" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160210    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c2a0744-9e32-4896-914e-dfb9173923e7" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160215    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a38c258-0d4e-4b70-afec-45f6025fdcc6" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160221    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c23612-28be-4e5f-aa95-71c40e82450e" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160226    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ddd1055-47a1-4c05-bd03-ff3c493f32a7" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160230    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="66fb1cac-4e6d-4d5a-812d-5054e1676769" containerName="maas-api"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160236    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="415f31e1-c808-4fb0-bb8c-33f928248892" containerName="payload-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160240    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca13cd9-66b8-456e-8147-5a951cc47d88" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160244    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c73d1b-a864-4534-944a-a543d04a2c3b" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160250    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="651ba59f-3f1f-479d-805e-30a17661c602" containerName="payload-pre-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160256    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="a848a44a-9e0a-47aa-811f-5a9930b9bd9f" containerName="payload-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160262    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="39539848-97ad-40d8-8082-d27a15143388" containerName="payload-processing"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160267    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="90d486b1-ecce-40a9-8353-04127a5111df" containerName="main"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160271    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b8551c-d7a1-4527-bd33-6f16c63ad713" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160276    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="3086a82d-b9ca-4c66-8b65-a1eb764909e5" containerName="istio-proxy"
Oct 30 20:14:17.160898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.160282    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="38fa73eb-88a7-4868-8e33-a407edc23aef" containerName="payload-pre-processing"
Oct 30 20:14:17.165021 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.164623    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.202830 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.202796    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ae2147-openshift-default-dockercfg-x9mn6\""
Oct 30 20:14:17.203055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.203011    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"]
Oct 30 20:14:17.316307 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.316268    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.316307 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.316310    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.316548 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.316404    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.316548 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.316432    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcvh5\" (UniqueName: \"kubernetes.io/projected/ea3c3516-eed1-4df6-a431-5aed37becda0-kube-api-access-bcvh5\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.316548 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.316451    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.316676 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.316592    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.316676 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.316646    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.316676 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.316667    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.316844 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.316766    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3c3516-eed1-4df6-a431-5aed37becda0-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.417928 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.417840    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.417928 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.417883    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.417928 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.417926    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.418257 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.417950    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcvh5\" (UniqueName: \"kubernetes.io/projected/ea3c3516-eed1-4df6-a431-5aed37becda0-kube-api-access-bcvh5\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.418257 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.417973    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.418257 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.418083    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.418257 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.418146    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.418257 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.418176    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.418257 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.418222    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3c3516-eed1-4df6-a431-5aed37becda0-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.418570 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.418341    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.418570 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.418380    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.418570 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.418408    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.418736 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.418659    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.418968 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.418943    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3c3516-eed1-4df6-a431-5aed37becda0-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.420338 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.420313    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.420576 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.420553    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.459410 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.459377    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcvh5\" (UniqueName: \"kubernetes.io/projected/ea3c3516-eed1-4df6-a431-5aed37becda0-kube-api-access-bcvh5\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.461326 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.461293    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.478703 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.478682    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:17.560372 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.560336    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"]
Oct 30 20:14:17.560597 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.560571    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj" podUID="f9581298-8aed-41b0-873c-10b0ffe14c27" containerName="istio-proxy" containerID="cri-o://02744e51d9d6add4e5b2da1eacd16e7d27b2cd056ed6feff6a960e7f0a3c453b" gracePeriod=30
Oct 30 20:14:17.849900 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.849856    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"]
Oct 30 20:14:17.856146 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.856106    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:14:17.856266 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.856197    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:14:17.856266 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:17.856255    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:14:18.751193 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:18.751153    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn" event={"ID":"ea3c3516-eed1-4df6-a431-5aed37becda0","Type":"ContainerStarted","Data":"54f9fadbd5309a7f6757485b24e8e811bb3a747fc974c2441c0ce52a4860580b"}
Oct 30 20:14:18.751844 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:18.751814    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn" event={"ID":"ea3c3516-eed1-4df6-a431-5aed37becda0","Type":"ContainerStarted","Data":"45ee5fa880a9f19e1309f978efc264c7aa32c3ffb6b92f116723942bbc8737fe"}
Oct 30 20:14:18.800869 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:18.800824    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn" podStartSLOduration=1.800809137 podStartE2EDuration="1.800809137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:18.798775141 +0000 UTC m=+2613.383044757" watchObservedRunningTime="2026-10-30 20:14:18.800809137 +0000 UTC m=+2613.385078749"
Oct 30 20:14:18.930675 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:18.930638    5408 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:14:18.930847 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:18.930697    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj" podUID="f9581298-8aed-41b0-873c-10b0ffe14c27" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:14:19.478955 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:19.478920    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:19.480335 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:19.480315    5408 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 20:14:19.480384 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:19.480360    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn" podUID="ea3c3516-eed1-4df6-a431-5aed37becda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 20:14:20.479903 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:20.479864    5408 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 20:14:20.480322 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:20.479925    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn" podUID="ea3c3516-eed1-4df6-a431-5aed37becda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 20:14:21.479686 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:21.479651    5408 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 20:14:21.479860 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:21.479717    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn" podUID="ea3c3516-eed1-4df6-a431-5aed37becda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 20:14:22.479529 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.479496    5408 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 20:14:22.479929 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.479549    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn" podUID="ea3c3516-eed1-4df6-a431-5aed37becda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 20:14:22.774008 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.773979    5408 generic.go:356] "Generic (PLEG): container finished" podID="f9581298-8aed-41b0-873c-10b0ffe14c27" containerID="02744e51d9d6add4e5b2da1eacd16e7d27b2cd056ed6feff6a960e7f0a3c453b" exitCode=0
Oct 30 20:14:22.774155 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.774123    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj" event={"ID":"f9581298-8aed-41b0-873c-10b0ffe14c27","Type":"ContainerDied","Data":"02744e51d9d6add4e5b2da1eacd16e7d27b2cd056ed6feff6a960e7f0a3c453b"}
Oct 30 20:14:22.878782 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.878759    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 20:14:22.972447 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.972415    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-podinfo\") pod \"f9581298-8aed-41b0-873c-10b0ffe14c27\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") "
Oct 30 20:14:22.972614 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.972460    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9581298-8aed-41b0-873c-10b0ffe14c27-kube-api-access-n9chj\" (UniqueName: \"kubernetes.io/projected/f9581298-8aed-41b0-873c-10b0ffe14c27-kube-api-access-n9chj\") pod \"f9581298-8aed-41b0-873c-10b0ffe14c27\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") "
Oct 30 20:14:22.972614 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.972494    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-workload-socket\") pod \"f9581298-8aed-41b0-873c-10b0ffe14c27\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") "
Oct 30 20:14:22.972614 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.972523    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-envoy\") pod \"f9581298-8aed-41b0-873c-10b0ffe14c27\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") "
Oct 30 20:14:22.972614 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.972578    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-data\") pod \"f9581298-8aed-41b0-873c-10b0ffe14c27\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") "
Oct 30 20:14:22.972846 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.972629    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9581298-8aed-41b0-873c-10b0ffe14c27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9581298-8aed-41b0-873c-10b0ffe14c27-istiod-ca-cert\") pod \"f9581298-8aed-41b0-873c-10b0ffe14c27\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") "
Oct 30 20:14:22.972846 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.972663    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-credential-socket\") pod \"f9581298-8aed-41b0-873c-10b0ffe14c27\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") "
Oct 30 20:14:22.972846 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.972707    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-token\" (UniqueName: \"kubernetes.io/projected/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-token\") pod \"f9581298-8aed-41b0-873c-10b0ffe14c27\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") "
Oct 30 20:14:22.972846 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.972749    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-workload-certs\") pod \"f9581298-8aed-41b0-873c-10b0ffe14c27\" (UID: \"f9581298-8aed-41b0-873c-10b0ffe14c27\") "
Oct 30 20:14:22.973021 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.972833    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-workload-socket" pod "f9581298-8aed-41b0-873c-10b0ffe14c27" (UID: "f9581298-8aed-41b0-873c-10b0ffe14c27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:22.973128 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.973092    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-workload-certs" pod "f9581298-8aed-41b0-873c-10b0ffe14c27" (UID: "f9581298-8aed-41b0-873c-10b0ffe14c27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:22.973180 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.973102    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-credential-socket" pod "f9581298-8aed-41b0-873c-10b0ffe14c27" (UID: "f9581298-8aed-41b0-873c-10b0ffe14c27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:22.973231 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.973202    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9581298-8aed-41b0-873c-10b0ffe14c27-istiod-ca-cert" pod "f9581298-8aed-41b0-873c-10b0ffe14c27" (UID: "f9581298-8aed-41b0-873c-10b0ffe14c27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:22.973231 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.973208    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:22.973231 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.973204    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-data" pod "f9581298-8aed-41b0-873c-10b0ffe14c27" (UID: "f9581298-8aed-41b0-873c-10b0ffe14c27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:22.973411 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.973235    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:22.973411 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.973246    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:22.974887 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.974855    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9581298-8aed-41b0-873c-10b0ffe14c27-kube-api-access-n9chj" pod "f9581298-8aed-41b0-873c-10b0ffe14c27" (UID: "f9581298-8aed-41b0-873c-10b0ffe14c27"). InnerVolumeSpecName "kube-api-access-n9chj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:22.974887 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.974855    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-envoy" pod "f9581298-8aed-41b0-873c-10b0ffe14c27" (UID: "f9581298-8aed-41b0-873c-10b0ffe14c27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:22.975098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.974925    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-token" pod "f9581298-8aed-41b0-873c-10b0ffe14c27" (UID: "f9581298-8aed-41b0-873c-10b0ffe14c27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:22.975098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:22.975024    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-podinfo" pod "f9581298-8aed-41b0-873c-10b0ffe14c27" (UID: "f9581298-8aed-41b0-873c-10b0ffe14c27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:14:23.073781 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:23.073702    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9chj\" (UniqueName: \"kubernetes.io/projected/f9581298-8aed-41b0-873c-10b0ffe14c27-kube-api-access-n9chj\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:23.073781 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:23.073732    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:23.073781 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:23.073742    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:23.073781 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:23.073751    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9581298-8aed-41b0-873c-10b0ffe14c27-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:23.073781 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:23.073759    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:23.073781 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:23.073767    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9581298-8aed-41b0-873c-10b0ffe14c27-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:23.480020 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:23.479932    5408 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 20:14:23.480020 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:23.479989    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn" podUID="ea3c3516-eed1-4df6-a431-5aed37becda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 20:14:23.780560 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:23.780531    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"
Oct 30 20:14:23.780560 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:23.780542    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj" event={"ID":"f9581298-8aed-41b0-873c-10b0ffe14c27","Type":"ContainerDied","Data":"15c2a91f1dc80da32e911b3097e592a885977e73066c8e2330fb610c834ca2d5"}
Oct 30 20:14:23.780800 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:23.780603    5408 scope.go:122] "RemoveContainer" containerID="02744e51d9d6add4e5b2da1eacd16e7d27b2cd056ed6feff6a960e7f0a3c453b"
Oct 30 20:14:23.816723 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:23.816694    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"]
Oct 30 20:14:23.855941 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:23.855916    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fjggj"]
Oct 30 20:14:24.483393 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:24.483364    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:24.483790 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:24.483654    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:24.484456 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:24.484439    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:14:25.532815 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:25.532776    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9581298-8aed-41b0-873c-10b0ffe14c27" path="/var/lib/kubelet/pods/f9581298-8aed-41b0-873c-10b0ffe14c27/volumes"
Oct 30 20:14:32.099208 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.099130    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"]
Oct 30 20:14:32.099622 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.099608    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f9581298-8aed-41b0-873c-10b0ffe14c27" containerName="istio-proxy"
Oct 30 20:14:32.099670 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.099625    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9581298-8aed-41b0-873c-10b0ffe14c27" containerName="istio-proxy"
Oct 30 20:14:32.099711 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.099697    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="50041f27-99a2-47ba-944a-9693459b8c80" containerName="istio-proxy"
Oct 30 20:14:32.099743 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.099714    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9581298-8aed-41b0-873c-10b0ffe14c27" containerName="istio-proxy"
Oct 30 20:14:32.104139 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.104123    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.125206 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.125179    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"]
Oct 30 20:14:32.142724 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.142698    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"]
Oct 30 20:14:32.146575 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.146549    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.156910 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.156881    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96b62769-3c47-447e-adbc-7683ef73f808-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.157174 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.157142    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96b62769-3c47-447e-adbc-7683ef73f808-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.157352 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.157333    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nmhz\" (UniqueName: \"kubernetes.io/projected/96b62769-3c47-447e-adbc-7683ef73f808-kube-api-access-7nmhz\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.157450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.157373    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.157450 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.157399    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.157570 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.157468    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.157570 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.157502    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.157570 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.157535    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96b62769-3c47-447e-adbc-7683ef73f808-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.157739 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.157567    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.159865 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.159843    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"]
Oct 30 20:14:32.258239 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.258204    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.258401 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.258249    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.258401 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.258283    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a99ad3-1536-4709-b53d-faa8de8ce056-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.258401 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.258342    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96b62769-3c47-447e-adbc-7683ef73f808-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.258527 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.258400    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.258565 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.258532    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.258565 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.258552    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.258631 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.258603    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96b62769-3c47-447e-adbc-7683ef73f808-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.258684 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.258668    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.258718 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.258693    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pfb8\" (UniqueName: \"kubernetes.io/projected/82a99ad3-1536-4709-b53d-faa8de8ce056-kube-api-access-9pfb8\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.258763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.258719    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.258763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.258740    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nmhz\" (UniqueName: \"kubernetes.io/projected/96b62769-3c47-447e-adbc-7683ef73f808-kube-api-access-7nmhz\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.258931 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.258914    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.258975 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.258941    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.259020 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.258974    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.259095 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.259075    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.259135 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.259103    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.259181 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.259135    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.259230 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.259205    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96b62769-3c47-447e-adbc-7683ef73f808-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.259288 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.259261    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.259421 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.259402    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.259522 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.259500    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.259796 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.259775    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96b62769-3c47-447e-adbc-7683ef73f808-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.260825 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.260807    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96b62769-3c47-447e-adbc-7683ef73f808-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.261223 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.261202    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.275712 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.275691    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96b62769-3c47-447e-adbc-7683ef73f808-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.280850 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.280832    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nmhz\" (UniqueName: \"kubernetes.io/projected/96b62769-3c47-447e-adbc-7683ef73f808-kube-api-access-7nmhz\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.359774 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.359697    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.359774 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.359746    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.359774 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.359767    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a99ad3-1536-4709-b53d-faa8de8ce056-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.360100 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.359803    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.360100 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.359833    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.360100 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.359863    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.360100 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.359887    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pfb8\" (UniqueName: \"kubernetes.io/projected/82a99ad3-1536-4709-b53d-faa8de8ce056-kube-api-access-9pfb8\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.360100 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.359923    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.360100 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.359963    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.360400 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.360204    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.360400 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.360389    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.360524 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.360411    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.360524 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.360458    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.360626 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.360608    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a99ad3-1536-4709-b53d-faa8de8ce056-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.362359 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.362328    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.362498 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.362481    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.380866 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.380843    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pfb8\" (UniqueName: \"kubernetes.io/projected/82a99ad3-1536-4709-b53d-faa8de8ce056-kube-api-access-9pfb8\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.380952 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.380868    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.416879 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.416853    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:32.461723 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.461671    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:32.703066 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.702969    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"]
Oct 30 20:14:32.706359 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:14:32.706334    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b62769_3c47_447e_adbc_7683ef73f808.slice/crio-28d75068965a139c7cb3f7d1c6836d0831a28e701ab8dbd4131b50a32346d63f WatchSource:0}: Error finding container 28d75068965a139c7cb3f7d1c6836d0831a28e701ab8dbd4131b50a32346d63f: Status 404 returned error can't find the container with id 28d75068965a139c7cb3f7d1c6836d0831a28e701ab8dbd4131b50a32346d63f
Oct 30 20:14:32.708886 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.708847    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:14:32.708968 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.708937    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:14:32.709008 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.708978    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:14:32.763233 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.763210    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"]
Oct 30 20:14:32.765221 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:14:32.765201    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a99ad3_1536_4709_b53d_faa8de8ce056.slice/crio-bd5f9ea88d4f471eea763e505bc0e15d72f78aad2fc67625249d1ed5eed24e5d WatchSource:0}: Error finding container bd5f9ea88d4f471eea763e505bc0e15d72f78aad2fc67625249d1ed5eed24e5d: Status 404 returned error can't find the container with id bd5f9ea88d4f471eea763e505bc0e15d72f78aad2fc67625249d1ed5eed24e5d
Oct 30 20:14:32.767941 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.767900    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:14:32.768066 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.767994    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:14:32.768118 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.768060    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:14:32.834190 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.834157    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd" event={"ID":"96b62769-3c47-447e-adbc-7683ef73f808","Type":"ContainerStarted","Data":"a140739f9e5819ab8d7911ad40c350c53583e4c117e2518dcde30b9227380e81"}
Oct 30 20:14:32.834365 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.834230    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd" event={"ID":"96b62769-3c47-447e-adbc-7683ef73f808","Type":"ContainerStarted","Data":"28d75068965a139c7cb3f7d1c6836d0831a28e701ab8dbd4131b50a32346d63f"}
Oct 30 20:14:32.836157 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.836133    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz" event={"ID":"82a99ad3-1536-4709-b53d-faa8de8ce056","Type":"ContainerStarted","Data":"610f7c93235f48a9009f0d7dbdbd017118b40537b0faeb0cf121c56cfc0a8a17"}
Oct 30 20:14:32.836279 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.836188    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz" event={"ID":"82a99ad3-1536-4709-b53d-faa8de8ce056","Type":"ContainerStarted","Data":"bd5f9ea88d4f471eea763e505bc0e15d72f78aad2fc67625249d1ed5eed24e5d"}
Oct 30 20:14:32.863489 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:32.863431    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd" podStartSLOduration=0.86341473 podStartE2EDuration="863.41473ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:32.861386879 +0000 UTC m=+2627.445656524" watchObservedRunningTime="2026-10-30 20:14:32.86341473 +0000 UTC m=+2627.447684349"
Oct 30 20:14:33.422055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:33.420017    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:33.422055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:33.420163    5408 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 20:14:33.422055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:33.420216    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd" podUID="96b62769-3c47-447e-adbc-7683ef73f808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 20:14:33.462292 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:33.462246    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:33.463599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:33.463570    5408 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 30 20:14:33.463733 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:33.463616    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz" podUID="82a99ad3-1536-4709-b53d-faa8de8ce056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 30 20:14:34.426055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:34.425479    5408 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 20:14:34.426055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:34.425545    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd" podUID="96b62769-3c47-447e-adbc-7683ef73f808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 20:14:34.462906 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:34.462873    5408 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 30 20:14:34.463068 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:34.462917    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz" podUID="82a99ad3-1536-4709-b53d-faa8de8ce056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 30 20:14:35.422905 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:35.422458    5408 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 20:14:35.422905 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:35.422525    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd" podUID="96b62769-3c47-447e-adbc-7683ef73f808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 20:14:35.463925 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:35.463521    5408 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 30 20:14:35.463925 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:35.463579    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz" podUID="82a99ad3-1536-4709-b53d-faa8de8ce056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 30 20:14:36.433054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:36.432259    5408 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 20:14:36.433054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:36.432321    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd" podUID="96b62769-3c47-447e-adbc-7683ef73f808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 20:14:36.463000 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:36.462956    5408 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 30 20:14:36.463194 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:36.463009    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz" podUID="82a99ad3-1536-4709-b53d-faa8de8ce056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 30 20:14:37.419524 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:37.419480    5408 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 20:14:37.420155 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:37.420125    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd" podUID="96b62769-3c47-447e-adbc-7683ef73f808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 20:14:37.462910 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:37.462877    5408 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 30 20:14:37.463151 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:37.463119    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz" podUID="82a99ad3-1536-4709-b53d-faa8de8ce056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 30 20:14:38.423167 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:38.423125    5408 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 20:14:38.423790 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:38.423752    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd" podUID="96b62769-3c47-447e-adbc-7683ef73f808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 20:14:38.462614 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:38.462576    5408 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 30 20:14:38.462794 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:38.462634    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz" podUID="82a99ad3-1536-4709-b53d-faa8de8ce056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 30 20:14:39.673837 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:14:39.673800    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:14:39.806739 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:39.804998    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:39.807658 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:39.807047    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:39.808577 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:39.808547    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:14:39.838262 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:39.838234    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:39.838584 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:39.838522    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:39.841301 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:39.841275    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:14:39.846381 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:14:39.846313    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz" podStartSLOduration=7.846299175 podStartE2EDuration="7.846299175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:32.89934812 +0000 UTC m=+2627.483617763" watchObservedRunningTime="2026-10-30 20:14:39.846299175 +0000 UTC m=+2634.430568792"
Oct 30 20:15:05.029866 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:05.029833    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2"]
Oct 30 20:15:05.035588 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:05.035564    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16c3b5db-vkrf2"]
Oct 30 20:15:05.530831 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:05.530799    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8ff1a51-6102-46df-9a93-cb2144161bf3" path="/var/lib/kubelet/pods/f8ff1a51-6102-46df-9a93-cb2144161bf3/volumes"
Oct 30 20:15:09.660021 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:15:09.659972    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:15:11.668055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:11.668010    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh"]
Oct 30 20:15:13.094517 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:13.094482    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"]
Oct 30 20:15:13.094936 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:13.094693    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm" podUID="bb98f97c-6cc1-4731-b3e8-e906dfb00e65" containerName="maas-api" containerID="cri-o://9118469e9b9bb832be8d3cea88b6f9c275905d486d5b0a424fdad585f17ffac9" gracePeriod=30
Oct 30 20:15:13.363414 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:13.363338    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"
Oct 30 20:15:13.435549 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:13.435504    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-metrics-certs\") pod \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\" (UID: \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\") "
Oct 30 20:15:13.435756 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:13.435690    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-openshift-service-ca\") pod \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\" (UID: \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\") "
Oct 30 20:15:13.435835 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:13.435755    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-maas-api-tls\") pod \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\" (UID: \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\") "
Oct 30 20:15:13.435898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:13.435856    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-kube-api-access-z5mrc\" (UniqueName: \"kubernetes.io/projected/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-kube-api-access-z5mrc\") pod \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\" (UID: \"bb98f97c-6cc1-4731-b3e8-e906dfb00e65\") "
Oct 30 20:15:13.436190 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:13.436158    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-openshift-service-ca" pod "bb98f97c-6cc1-4731-b3e8-e906dfb00e65" (UID: "bb98f97c-6cc1-4731-b3e8-e906dfb00e65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:13.436351 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:13.436334    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:13.438923 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:13.438850    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-kube-api-access-z5mrc" pod "bb98f97c-6cc1-4731-b3e8-e906dfb00e65" (UID: "bb98f97c-6cc1-4731-b3e8-e906dfb00e65"). InnerVolumeSpecName "kube-api-access-z5mrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:13.438923 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:13.438882    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-metrics-certs" pod "bb98f97c-6cc1-4731-b3e8-e906dfb00e65" (UID: "bb98f97c-6cc1-4731-b3e8-e906dfb00e65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:13.439909 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:13.439882    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-maas-api-tls" pod "bb98f97c-6cc1-4731-b3e8-e906dfb00e65" (UID: "bb98f97c-6cc1-4731-b3e8-e906dfb00e65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:13.538526 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:13.537695    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:13.538526 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:13.537724    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:13.538526 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:13.537741    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5mrc\" (UniqueName: \"kubernetes.io/projected/bb98f97c-6cc1-4731-b3e8-e906dfb00e65-kube-api-access-z5mrc\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:14.068174 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:14.068142    5408 generic.go:356] "Generic (PLEG): container finished" podID="bb98f97c-6cc1-4731-b3e8-e906dfb00e65" containerID="9118469e9b9bb832be8d3cea88b6f9c275905d486d5b0a424fdad585f17ffac9" exitCode=0
Oct 30 20:15:14.068369 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:14.068223    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"
Oct 30 20:15:14.068369 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:14.068226    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm" event={"ID":"bb98f97c-6cc1-4731-b3e8-e906dfb00e65","Type":"ContainerDied","Data":"9118469e9b9bb832be8d3cea88b6f9c275905d486d5b0a424fdad585f17ffac9"}
Oct 30 20:15:14.068369 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:14.068280    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm" event={"ID":"bb98f97c-6cc1-4731-b3e8-e906dfb00e65","Type":"ContainerDied","Data":"16cbbc852f4bcff508b4d1a4ce1d0f5168e707410d7e6fb5c2c9ca232eb8930d"}
Oct 30 20:15:14.068369 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:14.068308    5408 scope.go:122] "RemoveContainer" containerID="9118469e9b9bb832be8d3cea88b6f9c275905d486d5b0a424fdad585f17ffac9"
Oct 30 20:15:14.076818 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:14.076801    5408 scope.go:122] "RemoveContainer" containerID="9118469e9b9bb832be8d3cea88b6f9c275905d486d5b0a424fdad585f17ffac9"
Oct 30 20:15:14.077097 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:15:14.077077    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9118469e9b9bb832be8d3cea88b6f9c275905d486d5b0a424fdad585f17ffac9\": container with ID starting with 9118469e9b9bb832be8d3cea88b6f9c275905d486d5b0a424fdad585f17ffac9 not found: ID does not exist" containerID="9118469e9b9bb832be8d3cea88b6f9c275905d486d5b0a424fdad585f17ffac9"
Oct 30 20:15:14.077168 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:14.077109    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9118469e9b9bb832be8d3cea88b6f9c275905d486d5b0a424fdad585f17ffac9"} err="failed to get container status \"9118469e9b9bb832be8d3cea88b6f9c275905d486d5b0a424fdad585f17ffac9\": rpc error: code = NotFound desc = could not find container \"9118469e9b9bb832be8d3cea88b6f9c275905d486d5b0a424fdad585f17ffac9\": container with ID starting with 9118469e9b9bb832be8d3cea88b6f9c275905d486d5b0a424fdad585f17ffac9 not found: ID does not exist"
Oct 30 20:15:14.097441 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:14.097418    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"]
Oct 30 20:15:14.101908 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:14.101889    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-274922-8b8cb6d7f-xltkm"]
Oct 30 20:15:15.530668 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:15.530630    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb98f97c-6cc1-4731-b3e8-e906dfb00e65" path="/var/lib/kubelet/pods/bb98f97c-6cc1-4731-b3e8-e906dfb00e65/volumes"
Oct 30 20:15:16.669176 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:16.669130    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh" podUID="ce5e4fb9-3865-4f30-9978-5fb9a8600a14" containerName="payload-processing" containerID="cri-o://80d09244c3ca7b77b041f87cc65063d3292e97c3b9b32fb745320d4782ddd0ee" gracePeriod=25
Oct 30 20:15:17.515947 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:17.515914    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"]
Oct 30 20:15:17.516157 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:17.516134    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p" podUID="6bf03122-99f0-4878-98ef-eaa9325402f3" containerName="istio-proxy" containerID="cri-o://bbb0acf128e447b9edce4e90fc12c091805c88cda80f919cefc3a610511f99af" gracePeriod=30
Oct 30 20:15:20.232510 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:20.232477    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-mkl2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:15:20.234624 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:20.232530    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p" podUID="6bf03122-99f0-4878-98ef-eaa9325402f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:15:21.552267 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:21.552224    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh" podUID="ce5e4fb9-3865-4f30-9978-5fb9a8600a14" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:15:22.784165 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.784099    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 20:15:22.836489 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.836461    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-data\") pod \"6bf03122-99f0-4878-98ef-eaa9325402f3\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") "
Oct 30 20:15:22.836648 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.836512    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-token\" (UniqueName: \"kubernetes.io/projected/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-token\") pod \"6bf03122-99f0-4878-98ef-eaa9325402f3\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") "
Oct 30 20:15:22.836648 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.836554    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-podinfo\") pod \"6bf03122-99f0-4878-98ef-eaa9325402f3\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") "
Oct 30 20:15:22.836648 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.836591    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bf03122-99f0-4878-98ef-eaa9325402f3-kube-api-access-v4l67\" (UniqueName: \"kubernetes.io/projected/6bf03122-99f0-4878-98ef-eaa9325402f3-kube-api-access-v4l67\") pod \"6bf03122-99f0-4878-98ef-eaa9325402f3\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") "
Oct 30 20:15:22.836788 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.836747    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-data" pod "6bf03122-99f0-4878-98ef-eaa9325402f3" (UID: "6bf03122-99f0-4878-98ef-eaa9325402f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:22.836827 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.836806    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-envoy\") pod \"6bf03122-99f0-4878-98ef-eaa9325402f3\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") "
Oct 30 20:15:22.836875 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.836864    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-workload-certs\") pod \"6bf03122-99f0-4878-98ef-eaa9325402f3\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") "
Oct 30 20:15:22.836946 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.836924    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-credential-socket\") pod \"6bf03122-99f0-4878-98ef-eaa9325402f3\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") "
Oct 30 20:15:22.837107 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.836961    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bf03122-99f0-4878-98ef-eaa9325402f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf03122-99f0-4878-98ef-eaa9325402f3-istiod-ca-cert\") pod \"6bf03122-99f0-4878-98ef-eaa9325402f3\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") "
Oct 30 20:15:22.837107 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.837091    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-workload-socket\") pod \"6bf03122-99f0-4878-98ef-eaa9325402f3\" (UID: \"6bf03122-99f0-4878-98ef-eaa9325402f3\") "
Oct 30 20:15:22.837313 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.837241    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-workload-certs" pod "6bf03122-99f0-4878-98ef-eaa9325402f3" (UID: "6bf03122-99f0-4878-98ef-eaa9325402f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:22.837737 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.837521    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:22.837737 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.837546    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:22.837737 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.837672    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-workload-socket" pod "6bf03122-99f0-4878-98ef-eaa9325402f3" (UID: "6bf03122-99f0-4878-98ef-eaa9325402f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:22.837737 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.837702    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-credential-socket" pod "6bf03122-99f0-4878-98ef-eaa9325402f3" (UID: "6bf03122-99f0-4878-98ef-eaa9325402f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:22.837737 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.837715    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf03122-99f0-4878-98ef-eaa9325402f3-istiod-ca-cert" pod "6bf03122-99f0-4878-98ef-eaa9325402f3" (UID: "6bf03122-99f0-4878-98ef-eaa9325402f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:22.839315 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.839291    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-podinfo" pod "6bf03122-99f0-4878-98ef-eaa9325402f3" (UID: "6bf03122-99f0-4878-98ef-eaa9325402f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:15:22.839315 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.839302    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf03122-99f0-4878-98ef-eaa9325402f3-kube-api-access-v4l67" pod "6bf03122-99f0-4878-98ef-eaa9325402f3" (UID: "6bf03122-99f0-4878-98ef-eaa9325402f3"). InnerVolumeSpecName "kube-api-access-v4l67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:22.839474 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.839331    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-token" pod "6bf03122-99f0-4878-98ef-eaa9325402f3" (UID: "6bf03122-99f0-4878-98ef-eaa9325402f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:22.839474 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.839340    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-envoy" pod "6bf03122-99f0-4878-98ef-eaa9325402f3" (UID: "6bf03122-99f0-4878-98ef-eaa9325402f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:22.938560 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.938526    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:22.938560 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.938554    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:22.938560 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.938565    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf03122-99f0-4878-98ef-eaa9325402f3-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:22.938806 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.938574    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf03122-99f0-4878-98ef-eaa9325402f3-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:22.938806 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.938582    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:22.938806 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.938592    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf03122-99f0-4878-98ef-eaa9325402f3-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:22.938806 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:22.938602    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4l67\" (UniqueName: \"kubernetes.io/projected/6bf03122-99f0-4878-98ef-eaa9325402f3-kube-api-access-v4l67\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:23.116638 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:23.116556    5408 generic.go:356] "Generic (PLEG): container finished" podID="6bf03122-99f0-4878-98ef-eaa9325402f3" containerID="bbb0acf128e447b9edce4e90fc12c091805c88cda80f919cefc3a610511f99af" exitCode=0
Oct 30 20:15:23.116638 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:23.116628    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"
Oct 30 20:15:23.116893 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:23.116655    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p" event={"ID":"6bf03122-99f0-4878-98ef-eaa9325402f3","Type":"ContainerDied","Data":"bbb0acf128e447b9edce4e90fc12c091805c88cda80f919cefc3a610511f99af"}
Oct 30 20:15:23.116893 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:23.116714    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p" event={"ID":"6bf03122-99f0-4878-98ef-eaa9325402f3","Type":"ContainerDied","Data":"b305c5c572513b7932f88b84ea232569d31d213eb63ffdca8864787e664b71ae"}
Oct 30 20:15:23.116893 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:23.116745    5408 scope.go:122] "RemoveContainer" containerID="bbb0acf128e447b9edce4e90fc12c091805c88cda80f919cefc3a610511f99af"
Oct 30 20:15:23.128560 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:23.128532    5408 scope.go:122] "RemoveContainer" containerID="bbb0acf128e447b9edce4e90fc12c091805c88cda80f919cefc3a610511f99af"
Oct 30 20:15:23.128925 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:15:23.128902    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbb0acf128e447b9edce4e90fc12c091805c88cda80f919cefc3a610511f99af\": container with ID starting with bbb0acf128e447b9edce4e90fc12c091805c88cda80f919cefc3a610511f99af not found: ID does not exist" containerID="bbb0acf128e447b9edce4e90fc12c091805c88cda80f919cefc3a610511f99af"
Oct 30 20:15:23.129023 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:23.128936    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbb0acf128e447b9edce4e90fc12c091805c88cda80f919cefc3a610511f99af"} err="failed to get container status \"bbb0acf128e447b9edce4e90fc12c091805c88cda80f919cefc3a610511f99af\": rpc error: code = NotFound desc = could not find container \"bbb0acf128e447b9edce4e90fc12c091805c88cda80f919cefc3a610511f99af\": container with ID starting with bbb0acf128e447b9edce4e90fc12c091805c88cda80f919cefc3a610511f99af not found: ID does not exist"
Oct 30 20:15:23.143890 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:23.143864    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"]
Oct 30 20:15:23.148823 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:23.148800    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-mkl2p"]
Oct 30 20:15:23.530484 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:23.530446    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bf03122-99f0-4878-98ef-eaa9325402f3" path="/var/lib/kubelet/pods/6bf03122-99f0-4878-98ef-eaa9325402f3/volumes"
Oct 30 20:15:31.551925 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:31.551885    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh" podUID="ce5e4fb9-3865-4f30-9978-5fb9a8600a14" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:15:33.013870 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:33.013837    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf"]
Oct 30 20:15:33.033522 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:33.033492    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6"]
Oct 30 20:15:33.375134 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:33.368609    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"]
Oct 30 20:15:33.375563 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:33.375526    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx" podUID="d296b553-d243-4400-8c00-3de4f7b31ff7" containerName="maas-api" containerID="cri-o://4b9cd9f6fcbe1f7cf1a883f920a9b2256c6b17c6fb66f89745f15f226f0ea9b4" gracePeriod=30
Oct 30 20:15:33.708784 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:33.708758    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"
Oct 30 20:15:33.861629 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:33.861599    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d296b553-d243-4400-8c00-3de4f7b31ff7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d296b553-d243-4400-8c00-3de4f7b31ff7-metrics-certs\") pod \"d296b553-d243-4400-8c00-3de4f7b31ff7\" (UID: \"d296b553-d243-4400-8c00-3de4f7b31ff7\") "
Oct 30 20:15:33.861836 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:33.861659    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d296b553-d243-4400-8c00-3de4f7b31ff7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d296b553-d243-4400-8c00-3de4f7b31ff7-maas-api-tls\") pod \"d296b553-d243-4400-8c00-3de4f7b31ff7\" (UID: \"d296b553-d243-4400-8c00-3de4f7b31ff7\") "
Oct 30 20:15:33.861836 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:33.861700    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d296b553-d243-4400-8c00-3de4f7b31ff7-kube-api-access-jjwh8\" (UniqueName: \"kubernetes.io/projected/d296b553-d243-4400-8c00-3de4f7b31ff7-kube-api-access-jjwh8\") pod \"d296b553-d243-4400-8c00-3de4f7b31ff7\" (UID: \"d296b553-d243-4400-8c00-3de4f7b31ff7\") "
Oct 30 20:15:33.861836 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:33.861737    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d296b553-d243-4400-8c00-3de4f7b31ff7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d296b553-d243-4400-8c00-3de4f7b31ff7-openshift-service-ca\") pod \"d296b553-d243-4400-8c00-3de4f7b31ff7\" (UID: \"d296b553-d243-4400-8c00-3de4f7b31ff7\") "
Oct 30 20:15:33.862250 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:33.862215    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d296b553-d243-4400-8c00-3de4f7b31ff7-openshift-service-ca" pod "d296b553-d243-4400-8c00-3de4f7b31ff7" (UID: "d296b553-d243-4400-8c00-3de4f7b31ff7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:33.863862 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:33.863836    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d296b553-d243-4400-8c00-3de4f7b31ff7-maas-api-tls" pod "d296b553-d243-4400-8c00-3de4f7b31ff7" (UID: "d296b553-d243-4400-8c00-3de4f7b31ff7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:33.863964 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:33.863862    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d296b553-d243-4400-8c00-3de4f7b31ff7-kube-api-access-jjwh8" pod "d296b553-d243-4400-8c00-3de4f7b31ff7" (UID: "d296b553-d243-4400-8c00-3de4f7b31ff7"). InnerVolumeSpecName "kube-api-access-jjwh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:33.863964 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:33.863889    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d296b553-d243-4400-8c00-3de4f7b31ff7-metrics-certs" pod "d296b553-d243-4400-8c00-3de4f7b31ff7" (UID: "d296b553-d243-4400-8c00-3de4f7b31ff7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:33.963391 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:33.963319    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d296b553-d243-4400-8c00-3de4f7b31ff7-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:33.963391 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:33.963345    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d296b553-d243-4400-8c00-3de4f7b31ff7-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:33.963391 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:33.963356    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjwh8\" (UniqueName: \"kubernetes.io/projected/d296b553-d243-4400-8c00-3de4f7b31ff7-kube-api-access-jjwh8\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:33.963391 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:33.963366    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d296b553-d243-4400-8c00-3de4f7b31ff7-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:34.187315 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:34.187282    5408 generic.go:356] "Generic (PLEG): container finished" podID="d296b553-d243-4400-8c00-3de4f7b31ff7" containerID="4b9cd9f6fcbe1f7cf1a883f920a9b2256c6b17c6fb66f89745f15f226f0ea9b4" exitCode=0
Oct 30 20:15:34.187732 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:34.187342    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"
Oct 30 20:15:34.187732 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:34.187363    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx" event={"ID":"d296b553-d243-4400-8c00-3de4f7b31ff7","Type":"ContainerDied","Data":"4b9cd9f6fcbe1f7cf1a883f920a9b2256c6b17c6fb66f89745f15f226f0ea9b4"}
Oct 30 20:15:34.187732 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:34.187433    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx" event={"ID":"d296b553-d243-4400-8c00-3de4f7b31ff7","Type":"ContainerDied","Data":"26a92104976cf0e9b3a78062f8d778c0443a7f764bd8db30c3aca84e746b6faa"}
Oct 30 20:15:34.187732 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:34.187455    5408 scope.go:122] "RemoveContainer" containerID="4b9cd9f6fcbe1f7cf1a883f920a9b2256c6b17c6fb66f89745f15f226f0ea9b4"
Oct 30 20:15:34.197470 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:34.197451    5408 scope.go:122] "RemoveContainer" containerID="4b9cd9f6fcbe1f7cf1a883f920a9b2256c6b17c6fb66f89745f15f226f0ea9b4"
Oct 30 20:15:34.197741 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:15:34.197718    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b9cd9f6fcbe1f7cf1a883f920a9b2256c6b17c6fb66f89745f15f226f0ea9b4\": container with ID starting with 4b9cd9f6fcbe1f7cf1a883f920a9b2256c6b17c6fb66f89745f15f226f0ea9b4 not found: ID does not exist" containerID="4b9cd9f6fcbe1f7cf1a883f920a9b2256c6b17c6fb66f89745f15f226f0ea9b4"
Oct 30 20:15:34.197838 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:34.197747    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9cd9f6fcbe1f7cf1a883f920a9b2256c6b17c6fb66f89745f15f226f0ea9b4"} err="failed to get container status \"4b9cd9f6fcbe1f7cf1a883f920a9b2256c6b17c6fb66f89745f15f226f0ea9b4\": rpc error: code = NotFound desc = could not find container \"4b9cd9f6fcbe1f7cf1a883f920a9b2256c6b17c6fb66f89745f15f226f0ea9b4\": container with ID starting with 4b9cd9f6fcbe1f7cf1a883f920a9b2256c6b17c6fb66f89745f15f226f0ea9b4 not found: ID does not exist"
Oct 30 20:15:34.212749 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:34.212724    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"]
Oct 30 20:15:34.218387 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:34.218332    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae2147-657956b756-b65qx"]
Oct 30 20:15:35.531295 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:35.531266    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d296b553-d243-4400-8c00-3de4f7b31ff7" path="/var/lib/kubelet/pods/d296b553-d243-4400-8c00-3de4f7b31ff7/volumes"
Oct 30 20:15:36.915944 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:36.915921    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh"
Oct 30 20:15:36.992010 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:36.991782    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce5e4fb9-3865-4f30-9978-5fb9a8600a14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce5e4fb9-3865-4f30-9978-5fb9a8600a14-plugins-config-volume\") pod \"ce5e4fb9-3865-4f30-9978-5fb9a8600a14\" (UID: \"ce5e4fb9-3865-4f30-9978-5fb9a8600a14\") "
Oct 30 20:15:36.992010 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:36.991825    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce5e4fb9-3865-4f30-9978-5fb9a8600a14-kube-api-access-b8c8v\" (UniqueName: \"kubernetes.io/projected/ce5e4fb9-3865-4f30-9978-5fb9a8600a14-kube-api-access-b8c8v\") pod \"ce5e4fb9-3865-4f30-9978-5fb9a8600a14\" (UID: \"ce5e4fb9-3865-4f30-9978-5fb9a8600a14\") "
Oct 30 20:15:36.992576 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:36.992547    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce5e4fb9-3865-4f30-9978-5fb9a8600a14-plugins-config-volume" pod "ce5e4fb9-3865-4f30-9978-5fb9a8600a14" (UID: "ce5e4fb9-3865-4f30-9978-5fb9a8600a14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:36.994271 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:36.994246    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce5e4fb9-3865-4f30-9978-5fb9a8600a14-kube-api-access-b8c8v" pod "ce5e4fb9-3865-4f30-9978-5fb9a8600a14" (UID: "ce5e4fb9-3865-4f30-9978-5fb9a8600a14"). InnerVolumeSpecName "kube-api-access-b8c8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:37.093175 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:37.093145    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce5e4fb9-3865-4f30-9978-5fb9a8600a14-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:37.093175 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:37.093169    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8c8v\" (UniqueName: \"kubernetes.io/projected/ce5e4fb9-3865-4f30-9978-5fb9a8600a14-kube-api-access-b8c8v\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:37.219742 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:37.219704    5408 generic.go:356] "Generic (PLEG): container finished" podID="ce5e4fb9-3865-4f30-9978-5fb9a8600a14" containerID="80d09244c3ca7b77b041f87cc65063d3292e97c3b9b32fb745320d4782ddd0ee" exitCode=0
Oct 30 20:15:37.220191 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:37.220088    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh"
Oct 30 20:15:37.220439 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:37.220076    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh" event={"ID":"ce5e4fb9-3865-4f30-9978-5fb9a8600a14","Type":"ContainerDied","Data":"80d09244c3ca7b77b041f87cc65063d3292e97c3b9b32fb745320d4782ddd0ee"}
Oct 30 20:15:37.220595 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:37.220504    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh" event={"ID":"ce5e4fb9-3865-4f30-9978-5fb9a8600a14","Type":"ContainerDied","Data":"a14230449c3cec177853da7cfd8384e998a23af6e3d31e75f79e8a60056077d4"}
Oct 30 20:15:37.220595 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:37.220552    5408 scope.go:122] "RemoveContainer" containerID="80d09244c3ca7b77b041f87cc65063d3292e97c3b9b32fb745320d4782ddd0ee"
Oct 30 20:15:37.230455 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:37.230428    5408 scope.go:122] "RemoveContainer" containerID="80d09244c3ca7b77b041f87cc65063d3292e97c3b9b32fb745320d4782ddd0ee"
Oct 30 20:15:37.230759 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:15:37.230740    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d09244c3ca7b77b041f87cc65063d3292e97c3b9b32fb745320d4782ddd0ee\": container with ID starting with 80d09244c3ca7b77b041f87cc65063d3292e97c3b9b32fb745320d4782ddd0ee not found: ID does not exist" containerID="80d09244c3ca7b77b041f87cc65063d3292e97c3b9b32fb745320d4782ddd0ee"
Oct 30 20:15:37.230838 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:37.230771    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d09244c3ca7b77b041f87cc65063d3292e97c3b9b32fb745320d4782ddd0ee"} err="failed to get container status \"80d09244c3ca7b77b041f87cc65063d3292e97c3b9b32fb745320d4782ddd0ee\": rpc error: code = NotFound desc = could not find container \"80d09244c3ca7b77b041f87cc65063d3292e97c3b9b32fb745320d4782ddd0ee\": container with ID starting with 80d09244c3ca7b77b041f87cc65063d3292e97c3b9b32fb745320d4782ddd0ee not found: ID does not exist"
Oct 30 20:15:37.252555 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:37.252503    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh"]
Oct 30 20:15:37.259636 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:37.259617    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-274922-84f569749d-gnzfh"]
Oct 30 20:15:37.530124 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:37.530050    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce5e4fb9-3865-4f30-9978-5fb9a8600a14" path="/var/lib/kubelet/pods/ce5e4fb9-3865-4f30-9978-5fb9a8600a14/volumes"
Oct 30 20:15:38.014148 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:38.014113    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf" podUID="b623a795-c956-45ca-bf19-976ff116e46c" containerName="payload-processing" containerID="cri-o://9f74a27177e1c21edf2a3a25a9c34e8f0e4cff5134e7c81c739fabfade76b151" gracePeriod=25
Oct 30 20:15:38.034663 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:38.034628    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6" podUID="33ef1fdb-5a75-4de2-833b-ec7868412227" containerName="payload-pre-processing" containerID="cri-o://9a72686bced51c8467a6e1cfab8707f034a596b98a0092ddba2f51091f29c168" gracePeriod=25
Oct 30 20:15:39.659257 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:15:39.659222    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:15:42.481782 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:42.481705    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"]
Oct 30 20:15:42.482240 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:42.481918    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd" podUID="96b62769-3c47-447e-adbc-7683ef73f808" containerName="istio-proxy" containerID="cri-o://a140739f9e5819ab8d7911ad40c350c53583e4c117e2518dcde30b9227380e81" gracePeriod=30
Oct 30 20:15:42.483786 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:42.483759    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"]
Oct 30 20:15:42.484057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:42.483996    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn" podUID="ea3c3516-eed1-4df6-a431-5aed37becda0" containerName="istio-proxy" containerID="cri-o://54f9fadbd5309a7f6757485b24e8e811bb3a747fc974c2441c0ce52a4860580b" gracePeriod=30
Oct 30 20:15:42.511821 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:42.511791    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"]
Oct 30 20:15:42.511998 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:42.511978    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz" podUID="82a99ad3-1536-4709-b53d-faa8de8ce056" containerName="istio-proxy" containerID="cri-o://610f7c93235f48a9009f0d7dbdbd017118b40537b0faeb0cf121c56cfc0a8a17" gracePeriod=30
Oct 30 20:15:47.471934 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.471899    5408 scope.go:122] "RemoveContainer" containerID="b9a85ce8a978526993769918c905c099959169fba8327f31a3a5349550468d76"
Oct 30 20:15:47.473618 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.473588    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf" podUID="b623a795-c956-45ca-bf19-976ff116e46c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:15:47.473764 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.473661    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6" podUID="33ef1fdb-5a75-4de2-833b-ec7868412227" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:15:47.758703 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.758679    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:15:47.781199 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.781176    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:15:47.806439 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.806414    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3c3516-eed1-4df6-a431-5aed37becda0-kube-api-access-bcvh5\" (UniqueName: \"kubernetes.io/projected/ea3c3516-eed1-4df6-a431-5aed37becda0-kube-api-access-bcvh5\") pod \"ea3c3516-eed1-4df6-a431-5aed37becda0\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") "
Oct 30 20:15:47.806572 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.806505    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-workload-socket\") pod \"ea3c3516-eed1-4df6-a431-5aed37becda0\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") "
Oct 30 20:15:47.806572 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.806546    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-token\" (UniqueName: \"kubernetes.io/projected/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-token\") pod \"ea3c3516-eed1-4df6-a431-5aed37becda0\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") "
Oct 30 20:15:47.806709 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.806570    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-envoy\") pod \"ea3c3516-eed1-4df6-a431-5aed37becda0\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") "
Oct 30 20:15:47.806709 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.806635    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-credential-socket\") pod \"ea3c3516-eed1-4df6-a431-5aed37becda0\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") "
Oct 30 20:15:47.806709 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.806685    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-data\") pod \"ea3c3516-eed1-4df6-a431-5aed37becda0\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") "
Oct 30 20:15:47.806872 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.806715    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea3c3516-eed1-4df6-a431-5aed37becda0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3c3516-eed1-4df6-a431-5aed37becda0-istiod-ca-cert\") pod \"ea3c3516-eed1-4df6-a431-5aed37becda0\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") "
Oct 30 20:15:47.806872 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.806763    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-workload-certs\") pod \"ea3c3516-eed1-4df6-a431-5aed37becda0\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") "
Oct 30 20:15:47.806872 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.806813    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-podinfo\") pod \"ea3c3516-eed1-4df6-a431-5aed37becda0\" (UID: \"ea3c3516-eed1-4df6-a431-5aed37becda0\") "
Oct 30 20:15:47.806872 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.806825    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-workload-socket" pod "ea3c3516-eed1-4df6-a431-5aed37becda0" (UID: "ea3c3516-eed1-4df6-a431-5aed37becda0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.807452 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.807126    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-data" pod "ea3c3516-eed1-4df6-a431-5aed37becda0" (UID: "ea3c3516-eed1-4df6-a431-5aed37becda0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.807452 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.807294    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-credential-socket" pod "ea3c3516-eed1-4df6-a431-5aed37becda0" (UID: "ea3c3516-eed1-4df6-a431-5aed37becda0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.807452 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.807383    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-workload-certs" pod "ea3c3516-eed1-4df6-a431-5aed37becda0" (UID: "ea3c3516-eed1-4df6-a431-5aed37becda0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.807452 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.807391    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:47.807452 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.807429    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:47.807774 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.807462    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea3c3516-eed1-4df6-a431-5aed37becda0-istiod-ca-cert" pod "ea3c3516-eed1-4df6-a431-5aed37becda0" (UID: "ea3c3516-eed1-4df6-a431-5aed37becda0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:47.809484 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.809458    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-token" pod "ea3c3516-eed1-4df6-a431-5aed37becda0" (UID: "ea3c3516-eed1-4df6-a431-5aed37becda0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:47.809602 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.809462    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-envoy" pod "ea3c3516-eed1-4df6-a431-5aed37becda0" (UID: "ea3c3516-eed1-4df6-a431-5aed37becda0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.809602 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.809504    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-podinfo" pod "ea3c3516-eed1-4df6-a431-5aed37becda0" (UID: "ea3c3516-eed1-4df6-a431-5aed37becda0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:15:47.809602 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.809581    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3c3516-eed1-4df6-a431-5aed37becda0-kube-api-access-bcvh5" pod "ea3c3516-eed1-4df6-a431-5aed37becda0" (UID: "ea3c3516-eed1-4df6-a431-5aed37becda0"). InnerVolumeSpecName "kube-api-access-bcvh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:47.832825 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.832792    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:15:47.908830 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.908799    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b62769-3c47-447e-adbc-7683ef73f808-istio-token\" (UniqueName: \"kubernetes.io/projected/96b62769-3c47-447e-adbc-7683ef73f808-istio-token\") pod \"96b62769-3c47-447e-adbc-7683ef73f808\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") "
Oct 30 20:15:47.908999 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.908865    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82a99ad3-1536-4709-b53d-faa8de8ce056-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a99ad3-1536-4709-b53d-faa8de8ce056-istiod-ca-cert\") pod \"82a99ad3-1536-4709-b53d-faa8de8ce056\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") "
Oct 30 20:15:47.908999 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.908908    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-istio-envoy\") pod \"96b62769-3c47-447e-adbc-7683ef73f808\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") "
Oct 30 20:15:47.908999 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.908967    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-workload-socket\") pod \"82a99ad3-1536-4709-b53d-faa8de8ce056\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") "
Oct 30 20:15:47.909208 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.909003    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b62769-3c47-447e-adbc-7683ef73f808-kube-api-access-7nmhz\" (UniqueName: \"kubernetes.io/projected/96b62769-3c47-447e-adbc-7683ef73f808-kube-api-access-7nmhz\") pod \"96b62769-3c47-447e-adbc-7683ef73f808\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") "
Oct 30 20:15:47.909208 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.909052    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-token\" (UniqueName: \"kubernetes.io/projected/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-token\") pod \"82a99ad3-1536-4709-b53d-faa8de8ce056\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") "
Oct 30 20:15:47.909208 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.909100    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a99ad3-1536-4709-b53d-faa8de8ce056-kube-api-access-9pfb8\" (UniqueName: \"kubernetes.io/projected/82a99ad3-1536-4709-b53d-faa8de8ce056-kube-api-access-9pfb8\") pod \"82a99ad3-1536-4709-b53d-faa8de8ce056\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") "
Oct 30 20:15:47.909208 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.909153    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-workload-certs\") pod \"82a99ad3-1536-4709-b53d-faa8de8ce056\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") "
Oct 30 20:15:47.909428 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.909219    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-credential-socket\") pod \"96b62769-3c47-447e-adbc-7683ef73f808\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") "
Oct 30 20:15:47.909428 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.909253    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-workload-socket\") pod \"96b62769-3c47-447e-adbc-7683ef73f808\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") "
Oct 30 20:15:47.909428 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.909287    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-workload-certs\") pod \"96b62769-3c47-447e-adbc-7683ef73f808\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") "
Oct 30 20:15:47.909428 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.909329    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-podinfo\") pod \"82a99ad3-1536-4709-b53d-faa8de8ce056\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") "
Oct 30 20:15:47.909428 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.909325    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a99ad3-1536-4709-b53d-faa8de8ce056-istiod-ca-cert" pod "82a99ad3-1536-4709-b53d-faa8de8ce056" (UID: "82a99ad3-1536-4709-b53d-faa8de8ce056"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:47.909428 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.909371    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96b62769-3c47-447e-adbc-7683ef73f808-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96b62769-3c47-447e-adbc-7683ef73f808-istio-podinfo\") pod \"96b62769-3c47-447e-adbc-7683ef73f808\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") "
Oct 30 20:15:47.909428 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.909422    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-credential-socket\") pod \"82a99ad3-1536-4709-b53d-faa8de8ce056\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") "
Oct 30 20:15:47.909877 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.909453    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96b62769-3c47-447e-adbc-7683ef73f808-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96b62769-3c47-447e-adbc-7683ef73f808-istiod-ca-cert\") pod \"96b62769-3c47-447e-adbc-7683ef73f808\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") "
Oct 30 20:15:47.909877 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.909511    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-envoy\") pod \"82a99ad3-1536-4709-b53d-faa8de8ce056\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") "
Oct 30 20:15:47.909877 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.909545    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-data\") pod \"82a99ad3-1536-4709-b53d-faa8de8ce056\" (UID: \"82a99ad3-1536-4709-b53d-faa8de8ce056\") "
Oct 30 20:15:47.909877 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.909590    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-istio-data\") pod \"96b62769-3c47-447e-adbc-7683ef73f808\" (UID: \"96b62769-3c47-447e-adbc-7683ef73f808\") "
Oct 30 20:15:47.909877 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.909629    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-credential-socket" pod "96b62769-3c47-447e-adbc-7683ef73f808" (UID: "96b62769-3c47-447e-adbc-7683ef73f808"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.910196 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.910051    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a99ad3-1536-4709-b53d-faa8de8ce056-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:47.910196 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.910074    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:47.910196 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.910091    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcvh5\" (UniqueName: \"kubernetes.io/projected/ea3c3516-eed1-4df6-a431-5aed37becda0-kube-api-access-bcvh5\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:47.910196 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.910106    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:47.910196 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.910119    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:47.910196 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.910133    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:47.910196 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.910146    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:47.910196 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.910162    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3c3516-eed1-4df6-a431-5aed37becda0-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:47.910196 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.910177    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3c3516-eed1-4df6-a431-5aed37becda0-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:47.910683 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.910395    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-workload-socket" pod "82a99ad3-1536-4709-b53d-faa8de8ce056" (UID: "82a99ad3-1536-4709-b53d-faa8de8ce056"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.910683 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.910408    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-istio-data" pod "96b62769-3c47-447e-adbc-7683ef73f808" (UID: "96b62769-3c47-447e-adbc-7683ef73f808"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.910683 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.910466    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-workload-certs" pod "82a99ad3-1536-4709-b53d-faa8de8ce056" (UID: "82a99ad3-1536-4709-b53d-faa8de8ce056"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.910866 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.910843    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b62769-3c47-447e-adbc-7683ef73f808-istiod-ca-cert" pod "96b62769-3c47-447e-adbc-7683ef73f808" (UID: "96b62769-3c47-447e-adbc-7683ef73f808"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:47.911540 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.911119    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-credential-socket" pod "82a99ad3-1536-4709-b53d-faa8de8ce056" (UID: "82a99ad3-1536-4709-b53d-faa8de8ce056"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.911540 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.911280    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-workload-certs" pod "96b62769-3c47-447e-adbc-7683ef73f808" (UID: "96b62769-3c47-447e-adbc-7683ef73f808"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.911540 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.911397    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-data" pod "82a99ad3-1536-4709-b53d-faa8de8ce056" (UID: "82a99ad3-1536-4709-b53d-faa8de8ce056"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.911540 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.911514    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-workload-socket" pod "96b62769-3c47-447e-adbc-7683ef73f808" (UID: "96b62769-3c47-447e-adbc-7683ef73f808"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.912000 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.911943    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-token" pod "82a99ad3-1536-4709-b53d-faa8de8ce056" (UID: "82a99ad3-1536-4709-b53d-faa8de8ce056"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:47.912723 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.912696    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b62769-3c47-447e-adbc-7683ef73f808-istio-token" pod "96b62769-3c47-447e-adbc-7683ef73f808" (UID: "96b62769-3c47-447e-adbc-7683ef73f808"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:47.912877 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.912841    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-envoy" pod "82a99ad3-1536-4709-b53d-faa8de8ce056" (UID: "82a99ad3-1536-4709-b53d-faa8de8ce056"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.912954 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.912939    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-istio-envoy" pod "96b62769-3c47-447e-adbc-7683ef73f808" (UID: "96b62769-3c47-447e-adbc-7683ef73f808"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.913576 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.913552    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-podinfo" pod "82a99ad3-1536-4709-b53d-faa8de8ce056" (UID: "82a99ad3-1536-4709-b53d-faa8de8ce056"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:15:47.913667 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.913652    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a99ad3-1536-4709-b53d-faa8de8ce056-kube-api-access-9pfb8" pod "82a99ad3-1536-4709-b53d-faa8de8ce056" (UID: "82a99ad3-1536-4709-b53d-faa8de8ce056"). InnerVolumeSpecName "kube-api-access-9pfb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:47.913714 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.913654    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96b62769-3c47-447e-adbc-7683ef73f808-istio-podinfo" pod "96b62769-3c47-447e-adbc-7683ef73f808" (UID: "96b62769-3c47-447e-adbc-7683ef73f808"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:15:47.913994 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:47.913973    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b62769-3c47-447e-adbc-7683ef73f808-kube-api-access-7nmhz" pod "96b62769-3c47-447e-adbc-7683ef73f808" (UID: "96b62769-3c47-447e-adbc-7683ef73f808"). InnerVolumeSpecName "kube-api-access-7nmhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:48.011680 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.011644    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96b62769-3c47-447e-adbc-7683ef73f808-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.011680 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.011681    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.011854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.011698    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.011854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.011714    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nmhz\" (UniqueName: \"kubernetes.io/projected/96b62769-3c47-447e-adbc-7683ef73f808-kube-api-access-7nmhz\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.011854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.011732    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.011854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.011746    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pfb8\" (UniqueName: \"kubernetes.io/projected/82a99ad3-1536-4709-b53d-faa8de8ce056-kube-api-access-9pfb8\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.011854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.011762    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.011854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.011777    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.011854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.011791    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.011854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.011804    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.011854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.011818    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96b62769-3c47-447e-adbc-7683ef73f808-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.011854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.011834    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.011854 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.011848    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96b62769-3c47-447e-adbc-7683ef73f808-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.012260 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.011862    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.012260 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.011875    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a99ad3-1536-4709-b53d-faa8de8ce056-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.012260 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.011890    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96b62769-3c47-447e-adbc-7683ef73f808-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.275541 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.275463    5408 generic.go:356] "Generic (PLEG): container finished" podID="ea3c3516-eed1-4df6-a431-5aed37becda0" containerID="54f9fadbd5309a7f6757485b24e8e811bb3a747fc974c2441c0ce52a4860580b" exitCode=0
Oct 30 20:15:48.275541 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.275509    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"
Oct 30 20:15:48.275755 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.275542    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn" event={"ID":"ea3c3516-eed1-4df6-a431-5aed37becda0","Type":"ContainerDied","Data":"54f9fadbd5309a7f6757485b24e8e811bb3a747fc974c2441c0ce52a4860580b"}
Oct 30 20:15:48.275755 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.275601    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn" event={"ID":"ea3c3516-eed1-4df6-a431-5aed37becda0","Type":"ContainerDied","Data":"45ee5fa880a9f19e1309f978efc264c7aa32c3ffb6b92f116723942bbc8737fe"}
Oct 30 20:15:48.275755 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.275627    5408 scope.go:122] "RemoveContainer" containerID="54f9fadbd5309a7f6757485b24e8e811bb3a747fc974c2441c0ce52a4860580b"
Oct 30 20:15:48.277087 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.277064    5408 generic.go:356] "Generic (PLEG): container finished" podID="82a99ad3-1536-4709-b53d-faa8de8ce056" containerID="610f7c93235f48a9009f0d7dbdbd017118b40537b0faeb0cf121c56cfc0a8a17" exitCode=0
Oct 30 20:15:48.277231 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.277106    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"
Oct 30 20:15:48.277231 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.277149    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz" event={"ID":"82a99ad3-1536-4709-b53d-faa8de8ce056","Type":"ContainerDied","Data":"610f7c93235f48a9009f0d7dbdbd017118b40537b0faeb0cf121c56cfc0a8a17"}
Oct 30 20:15:48.277231 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.277204    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz" event={"ID":"82a99ad3-1536-4709-b53d-faa8de8ce056","Type":"ContainerDied","Data":"bd5f9ea88d4f471eea763e505bc0e15d72f78aad2fc67625249d1ed5eed24e5d"}
Oct 30 20:15:48.279115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.279089    5408 generic.go:356] "Generic (PLEG): container finished" podID="96b62769-3c47-447e-adbc-7683ef73f808" containerID="a140739f9e5819ab8d7911ad40c350c53583e4c117e2518dcde30b9227380e81" exitCode=0
Oct 30 20:15:48.279441 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.279148    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"
Oct 30 20:15:48.279622 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.279175    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd" event={"ID":"96b62769-3c47-447e-adbc-7683ef73f808","Type":"ContainerDied","Data":"a140739f9e5819ab8d7911ad40c350c53583e4c117e2518dcde30b9227380e81"}
Oct 30 20:15:48.279741 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.279694    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd" event={"ID":"96b62769-3c47-447e-adbc-7683ef73f808","Type":"ContainerDied","Data":"28d75068965a139c7cb3f7d1c6836d0831a28e701ab8dbd4131b50a32346d63f"}
Oct 30 20:15:48.303864 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.303771    5408 scope.go:122] "RemoveContainer" containerID="54f9fadbd5309a7f6757485b24e8e811bb3a747fc974c2441c0ce52a4860580b"
Oct 30 20:15:48.305156 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:15:48.305132    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f9fadbd5309a7f6757485b24e8e811bb3a747fc974c2441c0ce52a4860580b\": container with ID starting with 54f9fadbd5309a7f6757485b24e8e811bb3a747fc974c2441c0ce52a4860580b not found: ID does not exist" containerID="54f9fadbd5309a7f6757485b24e8e811bb3a747fc974c2441c0ce52a4860580b"
Oct 30 20:15:48.305258 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.305165    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f9fadbd5309a7f6757485b24e8e811bb3a747fc974c2441c0ce52a4860580b"} err="failed to get container status \"54f9fadbd5309a7f6757485b24e8e811bb3a747fc974c2441c0ce52a4860580b\": rpc error: code = NotFound desc = could not find container \"54f9fadbd5309a7f6757485b24e8e811bb3a747fc974c2441c0ce52a4860580b\": container with ID starting with 54f9fadbd5309a7f6757485b24e8e811bb3a747fc974c2441c0ce52a4860580b not found: ID does not exist"
Oct 30 20:15:48.305258 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.305188    5408 scope.go:122] "RemoveContainer" containerID="610f7c93235f48a9009f0d7dbdbd017118b40537b0faeb0cf121c56cfc0a8a17"
Oct 30 20:15:48.315206 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.315174    5408 scope.go:122] "RemoveContainer" containerID="610f7c93235f48a9009f0d7dbdbd017118b40537b0faeb0cf121c56cfc0a8a17"
Oct 30 20:15:48.315509 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:15:48.315484    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"610f7c93235f48a9009f0d7dbdbd017118b40537b0faeb0cf121c56cfc0a8a17\": container with ID starting with 610f7c93235f48a9009f0d7dbdbd017118b40537b0faeb0cf121c56cfc0a8a17 not found: ID does not exist" containerID="610f7c93235f48a9009f0d7dbdbd017118b40537b0faeb0cf121c56cfc0a8a17"
Oct 30 20:15:48.315593 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.315525    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"610f7c93235f48a9009f0d7dbdbd017118b40537b0faeb0cf121c56cfc0a8a17"} err="failed to get container status \"610f7c93235f48a9009f0d7dbdbd017118b40537b0faeb0cf121c56cfc0a8a17\": rpc error: code = NotFound desc = could not find container \"610f7c93235f48a9009f0d7dbdbd017118b40537b0faeb0cf121c56cfc0a8a17\": container with ID starting with 610f7c93235f48a9009f0d7dbdbd017118b40537b0faeb0cf121c56cfc0a8a17 not found: ID does not exist"
Oct 30 20:15:48.315593 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.315548    5408 scope.go:122] "RemoveContainer" containerID="a140739f9e5819ab8d7911ad40c350c53583e4c117e2518dcde30b9227380e81"
Oct 30 20:15:48.324230 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.324207    5408 scope.go:122] "RemoveContainer" containerID="a140739f9e5819ab8d7911ad40c350c53583e4c117e2518dcde30b9227380e81"
Oct 30 20:15:48.324785 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:15:48.324500    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a140739f9e5819ab8d7911ad40c350c53583e4c117e2518dcde30b9227380e81\": container with ID starting with a140739f9e5819ab8d7911ad40c350c53583e4c117e2518dcde30b9227380e81 not found: ID does not exist" containerID="a140739f9e5819ab8d7911ad40c350c53583e4c117e2518dcde30b9227380e81"
Oct 30 20:15:48.324893 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.324789    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a140739f9e5819ab8d7911ad40c350c53583e4c117e2518dcde30b9227380e81"} err="failed to get container status \"a140739f9e5819ab8d7911ad40c350c53583e4c117e2518dcde30b9227380e81\": rpc error: code = NotFound desc = could not find container \"a140739f9e5819ab8d7911ad40c350c53583e4c117e2518dcde30b9227380e81\": container with ID starting with a140739f9e5819ab8d7911ad40c350c53583e4c117e2518dcde30b9227380e81 not found: ID does not exist"
Oct 30 20:15:48.325844 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.325825    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"]
Oct 30 20:15:48.328634 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.328615    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-zg6dd"]
Oct 30 20:15:48.344642 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.344620    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"]
Oct 30 20:15:48.357176 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.357149    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-4nthz"]
Oct 30 20:15:48.371196 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.371171    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"]
Oct 30 20:15:48.380130 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:48.380104    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-hn6kn"]
Oct 30 20:15:49.529405 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:49.529374    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82a99ad3-1536-4709-b53d-faa8de8ce056" path="/var/lib/kubelet/pods/82a99ad3-1536-4709-b53d-faa8de8ce056/volumes"
Oct 30 20:15:49.529826 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:49.529812    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b62769-3c47-447e-adbc-7683ef73f808" path="/var/lib/kubelet/pods/96b62769-3c47-447e-adbc-7683ef73f808/volumes"
Oct 30 20:15:49.530229 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:49.530216    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea3c3516-eed1-4df6-a431-5aed37becda0" path="/var/lib/kubelet/pods/ea3c3516-eed1-4df6-a431-5aed37becda0/volumes"
Oct 30 20:15:57.473725 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:57.473635    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf" podUID="b623a795-c956-45ca-bf19-976ff116e46c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:15:57.473725 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:57.473657    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6" podUID="33ef1fdb-5a75-4de2-833b-ec7868412227" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:15:58.275100 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.275017    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf"
Oct 30 20:15:58.308477 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.308445    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6"
Oct 30 20:15:58.327058 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.327014    5408 generic.go:356] "Generic (PLEG): container finished" podID="b623a795-c956-45ca-bf19-976ff116e46c" containerID="9f74a27177e1c21edf2a3a25a9c34e8f0e4cff5134e7c81c739fabfade76b151" exitCode=0
Oct 30 20:15:58.327213 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.327058    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf" event={"ID":"b623a795-c956-45ca-bf19-976ff116e46c","Type":"ContainerDied","Data":"9f74a27177e1c21edf2a3a25a9c34e8f0e4cff5134e7c81c739fabfade76b151"}
Oct 30 20:15:58.327213 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.327077    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf"
Oct 30 20:15:58.327213 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.327111    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf" event={"ID":"b623a795-c956-45ca-bf19-976ff116e46c","Type":"ContainerDied","Data":"45d4c2cf1f59e9c4c35f05380287335f6fa8a37695d4649396d77ca2d4b53ab2"}
Oct 30 20:15:58.327213 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.327135    5408 scope.go:122] "RemoveContainer" containerID="9f74a27177e1c21edf2a3a25a9c34e8f0e4cff5134e7c81c739fabfade76b151"
Oct 30 20:15:58.328761 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.328738    5408 generic.go:356] "Generic (PLEG): container finished" podID="33ef1fdb-5a75-4de2-833b-ec7868412227" containerID="9a72686bced51c8467a6e1cfab8707f034a596b98a0092ddba2f51091f29c168" exitCode=0
Oct 30 20:15:58.328894 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.328767    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6" event={"ID":"33ef1fdb-5a75-4de2-833b-ec7868412227","Type":"ContainerDied","Data":"9a72686bced51c8467a6e1cfab8707f034a596b98a0092ddba2f51091f29c168"}
Oct 30 20:15:58.328894 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.328786    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6"
Oct 30 20:15:58.328894 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.328803    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6" event={"ID":"33ef1fdb-5a75-4de2-833b-ec7868412227","Type":"ContainerDied","Data":"27a7b15b1b62097954e0ef23f0547fa585346ead9cdff0ed234b8d6e22ec0c6c"}
Oct 30 20:15:58.337262 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.337242    5408 scope.go:122] "RemoveContainer" containerID="9f74a27177e1c21edf2a3a25a9c34e8f0e4cff5134e7c81c739fabfade76b151"
Oct 30 20:15:58.337558 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:15:58.337535    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f74a27177e1c21edf2a3a25a9c34e8f0e4cff5134e7c81c739fabfade76b151\": container with ID starting with 9f74a27177e1c21edf2a3a25a9c34e8f0e4cff5134e7c81c739fabfade76b151 not found: ID does not exist" containerID="9f74a27177e1c21edf2a3a25a9c34e8f0e4cff5134e7c81c739fabfade76b151"
Oct 30 20:15:58.337639 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.337569    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f74a27177e1c21edf2a3a25a9c34e8f0e4cff5134e7c81c739fabfade76b151"} err="failed to get container status \"9f74a27177e1c21edf2a3a25a9c34e8f0e4cff5134e7c81c739fabfade76b151\": rpc error: code = NotFound desc = could not find container \"9f74a27177e1c21edf2a3a25a9c34e8f0e4cff5134e7c81c739fabfade76b151\": container with ID starting with 9f74a27177e1c21edf2a3a25a9c34e8f0e4cff5134e7c81c739fabfade76b151 not found: ID does not exist"
Oct 30 20:15:58.337639 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.337592    5408 scope.go:122] "RemoveContainer" containerID="9a72686bced51c8467a6e1cfab8707f034a596b98a0092ddba2f51091f29c168"
Oct 30 20:15:58.346811 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.346792    5408 scope.go:122] "RemoveContainer" containerID="9a72686bced51c8467a6e1cfab8707f034a596b98a0092ddba2f51091f29c168"
Oct 30 20:15:58.347099 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:15:58.347078    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a72686bced51c8467a6e1cfab8707f034a596b98a0092ddba2f51091f29c168\": container with ID starting with 9a72686bced51c8467a6e1cfab8707f034a596b98a0092ddba2f51091f29c168 not found: ID does not exist" containerID="9a72686bced51c8467a6e1cfab8707f034a596b98a0092ddba2f51091f29c168"
Oct 30 20:15:58.347166 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.347105    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a72686bced51c8467a6e1cfab8707f034a596b98a0092ddba2f51091f29c168"} err="failed to get container status \"9a72686bced51c8467a6e1cfab8707f034a596b98a0092ddba2f51091f29c168\": rpc error: code = NotFound desc = could not find container \"9a72686bced51c8467a6e1cfab8707f034a596b98a0092ddba2f51091f29c168\": container with ID starting with 9a72686bced51c8467a6e1cfab8707f034a596b98a0092ddba2f51091f29c168 not found: ID does not exist"
Oct 30 20:15:58.421077 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.421053    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b623a795-c956-45ca-bf19-976ff116e46c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b623a795-c956-45ca-bf19-976ff116e46c-plugins-config-volume\") pod \"b623a795-c956-45ca-bf19-976ff116e46c\" (UID: \"b623a795-c956-45ca-bf19-976ff116e46c\") "
Oct 30 20:15:58.421210 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.421112    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ef1fdb-5a75-4de2-833b-ec7868412227-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ef1fdb-5a75-4de2-833b-ec7868412227-plugins-config-volume\") pod \"33ef1fdb-5a75-4de2-833b-ec7868412227\" (UID: \"33ef1fdb-5a75-4de2-833b-ec7868412227\") "
Oct 30 20:15:58.421210 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.421150    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b623a795-c956-45ca-bf19-976ff116e46c-kube-api-access-69k8w\" (UniqueName: \"kubernetes.io/projected/b623a795-c956-45ca-bf19-976ff116e46c-kube-api-access-69k8w\") pod \"b623a795-c956-45ca-bf19-976ff116e46c\" (UID: \"b623a795-c956-45ca-bf19-976ff116e46c\") "
Oct 30 20:15:58.421333 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.421209    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ef1fdb-5a75-4de2-833b-ec7868412227-kube-api-access-r9lhm\" (UniqueName: \"kubernetes.io/projected/33ef1fdb-5a75-4de2-833b-ec7868412227-kube-api-access-r9lhm\") pod \"33ef1fdb-5a75-4de2-833b-ec7868412227\" (UID: \"33ef1fdb-5a75-4de2-833b-ec7868412227\") "
Oct 30 20:15:58.421436 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.421416    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ef1fdb-5a75-4de2-833b-ec7868412227-plugins-config-volume" pod "33ef1fdb-5a75-4de2-833b-ec7868412227" (UID: "33ef1fdb-5a75-4de2-833b-ec7868412227"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:58.421497 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.421460    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b623a795-c956-45ca-bf19-976ff116e46c-plugins-config-volume" pod "b623a795-c956-45ca-bf19-976ff116e46c" (UID: "b623a795-c956-45ca-bf19-976ff116e46c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:58.421544 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.421506    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ef1fdb-5a75-4de2-833b-ec7868412227-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:58.423147 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.423125    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b623a795-c956-45ca-bf19-976ff116e46c-kube-api-access-69k8w" pod "b623a795-c956-45ca-bf19-976ff116e46c" (UID: "b623a795-c956-45ca-bf19-976ff116e46c"). InnerVolumeSpecName "kube-api-access-69k8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:58.423301 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.423281    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ef1fdb-5a75-4de2-833b-ec7868412227-kube-api-access-r9lhm" pod "33ef1fdb-5a75-4de2-833b-ec7868412227" (UID: "33ef1fdb-5a75-4de2-833b-ec7868412227"). InnerVolumeSpecName "kube-api-access-r9lhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:58.522176 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.522149    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69k8w\" (UniqueName: \"kubernetes.io/projected/b623a795-c956-45ca-bf19-976ff116e46c-kube-api-access-69k8w\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:58.522176 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.522174    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9lhm\" (UniqueName: \"kubernetes.io/projected/33ef1fdb-5a75-4de2-833b-ec7868412227-kube-api-access-r9lhm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:58.522505 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.522188    5408 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b623a795-c956-45ca-bf19-976ff116e46c-plugins-config-volume\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:58.665964 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.665934    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf"]
Oct 30 20:15:58.673763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.673736    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae2147-86646cf68d-vdmqf"]
Oct 30 20:15:58.698455 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.698432    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6"]
Oct 30 20:15:58.703819 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:58.703797    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae2147-5bbcf7cf9-8ksm6"]
Oct 30 20:15:59.529144 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:59.529112    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ef1fdb-5a75-4de2-833b-ec7868412227" path="/var/lib/kubelet/pods/33ef1fdb-5a75-4de2-833b-ec7868412227/volumes"
Oct 30 20:15:59.529518 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:15:59.529502    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b623a795-c956-45ca-bf19-976ff116e46c" path="/var/lib/kubelet/pods/b623a795-c956-45ca-bf19-976ff116e46c/volumes"
Oct 30 20:16:09.384521 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:09.384486    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"]
Oct 30 20:16:09.385251 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:09.384712    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx" podUID="ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51" containerName="maas-api" containerID="cri-o://2e009b201eefc50928bf0987e06f76d98868f454d4a3748229a01c9cb9ae34f5" gracePeriod=30
Oct 30 20:16:09.657430 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:09.653496    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"
Oct 30 20:16:09.697917 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:16:09.697780    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:16:09.844120 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:09.844091    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-kube-api-access-hcvsc\" (UniqueName: \"kubernetes.io/projected/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-kube-api-access-hcvsc\") pod \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\" (UID: \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\") "
Oct 30 20:16:09.844296 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:09.844188    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-openshift-service-ca\") pod \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\" (UID: \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\") "
Oct 30 20:16:09.844360 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:09.844340    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-metrics-certs\") pod \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\" (UID: \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\") "
Oct 30 20:16:09.844400 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:09.844382    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-maas-api-tls\") pod \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\" (UID: \"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51\") "
Oct 30 20:16:09.844527 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:09.844499    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-openshift-service-ca" pod "ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51" (UID: "ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:09.844672 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:09.844657    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:09.846505 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:09.846478    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-metrics-certs" pod "ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51" (UID: "ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:09.846652 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:09.846501    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-maas-api-tls" pod "ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51" (UID: "ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:09.846652 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:09.846610    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-kube-api-access-hcvsc" pod "ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51" (UID: "ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51"). InnerVolumeSpecName "kube-api-access-hcvsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:09.945861 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:09.945798    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:09.945861 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:09.945824    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:09.945861 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:09.945834    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcvsc\" (UniqueName: \"kubernetes.io/projected/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51-kube-api-access-hcvsc\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:10.387830 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:10.387797    5408 generic.go:356] "Generic (PLEG): container finished" podID="ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51" containerID="2e009b201eefc50928bf0987e06f76d98868f454d4a3748229a01c9cb9ae34f5" exitCode=0
Oct 30 20:16:10.388349 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:10.387838    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"
Oct 30 20:16:10.388349 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:10.387883    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx" event={"ID":"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51","Type":"ContainerDied","Data":"2e009b201eefc50928bf0987e06f76d98868f454d4a3748229a01c9cb9ae34f5"}
Oct 30 20:16:10.388349 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:10.387935    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx" event={"ID":"ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51","Type":"ContainerDied","Data":"f6da300af388c88f79b2b7c507bc58261e6b50f5081759340d1fd397a63e0a25"}
Oct 30 20:16:10.388349 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:10.387958    5408 scope.go:122] "RemoveContainer" containerID="2e009b201eefc50928bf0987e06f76d98868f454d4a3748229a01c9cb9ae34f5"
Oct 30 20:16:10.397385 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:10.397368    5408 scope.go:122] "RemoveContainer" containerID="2e009b201eefc50928bf0987e06f76d98868f454d4a3748229a01c9cb9ae34f5"
Oct 30 20:16:10.397651 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:16:10.397613    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e009b201eefc50928bf0987e06f76d98868f454d4a3748229a01c9cb9ae34f5\": container with ID starting with 2e009b201eefc50928bf0987e06f76d98868f454d4a3748229a01c9cb9ae34f5 not found: ID does not exist" containerID="2e009b201eefc50928bf0987e06f76d98868f454d4a3748229a01c9cb9ae34f5"
Oct 30 20:16:10.397651 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:10.397641    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e009b201eefc50928bf0987e06f76d98868f454d4a3748229a01c9cb9ae34f5"} err="failed to get container status \"2e009b201eefc50928bf0987e06f76d98868f454d4a3748229a01c9cb9ae34f5\": rpc error: code = NotFound desc = could not find container \"2e009b201eefc50928bf0987e06f76d98868f454d4a3748229a01c9cb9ae34f5\": container with ID starting with 2e009b201eefc50928bf0987e06f76d98868f454d4a3748229a01c9cb9ae34f5 not found: ID does not exist"
Oct 30 20:16:10.410629 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:10.410600    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"]
Oct 30 20:16:10.414944 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:10.414924    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a8207-7d877b4d86-82cgx"]
Oct 30 20:16:11.529911 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:11.529869    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51" path="/var/lib/kubelet/pods/ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51/volumes"
Oct 30 20:16:18.441409 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:18.441375    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"]
Oct 30 20:16:18.441841 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:18.441437    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"]
Oct 30 20:16:18.441841 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:18.441605    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs" podUID="ad4e73d4-d56f-499f-ac82-01e963174b4d" containerName="istio-proxy" containerID="cri-o://5e9066a512fc0cac12e38031ad1e2f13d09b1cc6575a432eaaf83a282af39c11" gracePeriod=30
Oct 30 20:16:18.441841 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:18.441775    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" podUID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" containerName="istio-proxy" containerID="cri-o://55539d73b315aa3a84ad5975871bbfde61277cd7de8088e65702be91617d8a57" gracePeriod=30
Oct 30 20:16:21.209663 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:21.209625    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6df954cb67-w57gl"]
Oct 30 20:16:21.210181 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:21.209871    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6df954cb67-w57gl" podUID="2e480f59-6b71-4efe-905f-b1959a4a3891" containerName="manager" containerID="cri-o://14144c57a3c7c6fb5b19d7d8838f35fb6f4575bd453f2bf814b3a6fb059457da" gracePeriod=30
Oct 30 20:16:21.446330 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:21.446298    5408 generic.go:356] "Generic (PLEG): container finished" podID="2e480f59-6b71-4efe-905f-b1959a4a3891" containerID="14144c57a3c7c6fb5b19d7d8838f35fb6f4575bd453f2bf814b3a6fb059457da" exitCode=0
Oct 30 20:16:21.446506 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:21.446387    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6df954cb67-w57gl" event={"ID":"2e480f59-6b71-4efe-905f-b1959a4a3891","Type":"ContainerDied","Data":"14144c57a3c7c6fb5b19d7d8838f35fb6f4575bd453f2bf814b3a6fb059457da"}
Oct 30 20:16:21.469589 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:21.469535    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6df954cb67-w57gl"
Oct 30 20:16:21.547764 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:21.547735    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e480f59-6b71-4efe-905f-b1959a4a3891-kube-api-access-bvp7s\" (UniqueName: \"kubernetes.io/projected/2e480f59-6b71-4efe-905f-b1959a4a3891-kube-api-access-bvp7s\") pod \"2e480f59-6b71-4efe-905f-b1959a4a3891\" (UID: \"2e480f59-6b71-4efe-905f-b1959a4a3891\") "
Oct 30 20:16:21.547941 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:21.547803    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e480f59-6b71-4efe-905f-b1959a4a3891-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e480f59-6b71-4efe-905f-b1959a4a3891-metrics-certs\") pod \"2e480f59-6b71-4efe-905f-b1959a4a3891\" (UID: \"2e480f59-6b71-4efe-905f-b1959a4a3891\") "
Oct 30 20:16:21.547941 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:21.547852    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e480f59-6b71-4efe-905f-b1959a4a3891-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e480f59-6b71-4efe-905f-b1959a4a3891-webhook-cert\") pod \"2e480f59-6b71-4efe-905f-b1959a4a3891\" (UID: \"2e480f59-6b71-4efe-905f-b1959a4a3891\") "
Oct 30 20:16:21.549968 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:21.549937    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e480f59-6b71-4efe-905f-b1959a4a3891-webhook-cert" pod "2e480f59-6b71-4efe-905f-b1959a4a3891" (UID: "2e480f59-6b71-4efe-905f-b1959a4a3891"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:21.550127 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:21.549968    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e480f59-6b71-4efe-905f-b1959a4a3891-metrics-certs" pod "2e480f59-6b71-4efe-905f-b1959a4a3891" (UID: "2e480f59-6b71-4efe-905f-b1959a4a3891"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:21.550127 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:21.549996    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e480f59-6b71-4efe-905f-b1959a4a3891-kube-api-access-bvp7s" pod "2e480f59-6b71-4efe-905f-b1959a4a3891" (UID: "2e480f59-6b71-4efe-905f-b1959a4a3891"). InnerVolumeSpecName "kube-api-access-bvp7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:21.649278 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:21.649243    5408 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e480f59-6b71-4efe-905f-b1959a4a3891-webhook-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:21.649278 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:21.649276    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvp7s\" (UniqueName: \"kubernetes.io/projected/2e480f59-6b71-4efe-905f-b1959a4a3891-kube-api-access-bvp7s\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:21.649278 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:21.649286    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e480f59-6b71-4efe-905f-b1959a4a3891-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:22.453076 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:22.453025    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6df954cb67-w57gl"
Oct 30 20:16:22.453622 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:22.453050    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6df954cb67-w57gl" event={"ID":"2e480f59-6b71-4efe-905f-b1959a4a3891","Type":"ContainerDied","Data":"4ed654cb2e73897bdacdd4bfe27d146a4085d47ff167af8acee8993faeb6f4f5"}
Oct 30 20:16:22.453622 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:22.453176    5408 scope.go:122] "RemoveContainer" containerID="14144c57a3c7c6fb5b19d7d8838f35fb6f4575bd453f2bf814b3a6fb059457da"
Oct 30 20:16:22.480427 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:22.480398    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6df954cb67-w57gl"]
Oct 30 20:16:22.487151 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:22.487130    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6df954cb67-w57gl"]
Oct 30 20:16:23.182902 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.182858    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6df954cb67-kr7xq"]
Oct 30 20:16:23.183455 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183432    5408 cpu_manager.go:414] "containerMap: removing container" podUID="ce5e4fb9-3865-4f30-9978-5fb9a8600a14" containerName="payload-processing"
Oct 30 20:16:23.183552 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183456    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce5e4fb9-3865-4f30-9978-5fb9a8600a14" containerName="payload-processing"
Oct 30 20:16:23.183552 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183471    5408 cpu_manager.go:414] "containerMap: removing container" podUID="bb98f97c-6cc1-4731-b3e8-e906dfb00e65" containerName="maas-api"
Oct 30 20:16:23.183552 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183480    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb98f97c-6cc1-4731-b3e8-e906dfb00e65" containerName="maas-api"
Oct 30 20:16:23.183552 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183496    5408 cpu_manager.go:414] "containerMap: removing container" podUID="96b62769-3c47-447e-adbc-7683ef73f808" containerName="istio-proxy"
Oct 30 20:16:23.183552 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183504    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b62769-3c47-447e-adbc-7683ef73f808" containerName="istio-proxy"
Oct 30 20:16:23.183552 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183518    5408 cpu_manager.go:414] "containerMap: removing container" podUID="2e480f59-6b71-4efe-905f-b1959a4a3891" containerName="manager"
Oct 30 20:16:23.183552 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183526    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e480f59-6b71-4efe-905f-b1959a4a3891" containerName="manager"
Oct 30 20:16:23.183552 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183544    5408 cpu_manager.go:414] "containerMap: removing container" podUID="82a99ad3-1536-4709-b53d-faa8de8ce056" containerName="istio-proxy"
Oct 30 20:16:23.183552 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183552    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a99ad3-1536-4709-b53d-faa8de8ce056" containerName="istio-proxy"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183563    5408 cpu_manager.go:414] "containerMap: removing container" podUID="d296b553-d243-4400-8c00-3de4f7b31ff7" containerName="maas-api"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183570    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d296b553-d243-4400-8c00-3de4f7b31ff7" containerName="maas-api"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183590    5408 cpu_manager.go:414] "containerMap: removing container" podUID="6bf03122-99f0-4878-98ef-eaa9325402f3" containerName="istio-proxy"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183597    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf03122-99f0-4878-98ef-eaa9325402f3" containerName="istio-proxy"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183607    5408 cpu_manager.go:414] "containerMap: removing container" podUID="ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51" containerName="maas-api"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183613    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51" containerName="maas-api"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183630    5408 cpu_manager.go:414] "containerMap: removing container" podUID="b623a795-c956-45ca-bf19-976ff116e46c" containerName="payload-processing"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183637    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b623a795-c956-45ca-bf19-976ff116e46c" containerName="payload-processing"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183653    5408 cpu_manager.go:414] "containerMap: removing container" podUID="33ef1fdb-5a75-4de2-833b-ec7868412227" containerName="payload-pre-processing"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183662    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ef1fdb-5a75-4de2-833b-ec7868412227" containerName="payload-pre-processing"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183678    5408 cpu_manager.go:414] "containerMap: removing container" podUID="ea3c3516-eed1-4df6-a431-5aed37becda0" containerName="istio-proxy"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183685    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3c3516-eed1-4df6-a431-5aed37becda0" containerName="istio-proxy"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183765    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf03122-99f0-4878-98ef-eaa9325402f3" containerName="istio-proxy"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183777    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed52fa8d-f9f6-4b79-bdf5-32af6f7b9b51" containerName="maas-api"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183789    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="b623a795-c956-45ca-bf19-976ff116e46c" containerName="payload-processing"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183799    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ef1fdb-5a75-4de2-833b-ec7868412227" containerName="payload-pre-processing"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183809    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3c3516-eed1-4df6-a431-5aed37becda0" containerName="istio-proxy"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183820    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce5e4fb9-3865-4f30-9978-5fb9a8600a14" containerName="payload-processing"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183827    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb98f97c-6cc1-4731-b3e8-e906dfb00e65" containerName="maas-api"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183837    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b62769-3c47-447e-adbc-7683ef73f808" containerName="istio-proxy"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183845    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e480f59-6b71-4efe-905f-b1959a4a3891" containerName="manager"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183854    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a99ad3-1536-4709-b53d-faa8de8ce056" containerName="istio-proxy"
Oct 30 20:16:23.184042 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.183863    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="d296b553-d243-4400-8c00-3de4f7b31ff7" containerName="maas-api"
Oct 30 20:16:23.197050 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.188614    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6df954cb67-kr7xq"
Oct 30 20:16:23.198181 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.197686    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 20:16:23.201065 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.199215    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2fq9m\""
Oct 30 20:16:23.201065 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.199303    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 20:16:23.201065 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.199362    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 20:16:23.201536 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.201504    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6df954cb67-kr7xq"]
Oct 30 20:16:23.263724 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.263467    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5fv4\" (UniqueName: \"kubernetes.io/projected/f638007b-aeae-48e2-a158-fb3e4fdff655-kube-api-access-s5fv4\") pod \"maas-controller-6df954cb67-kr7xq\" (UID: \"f638007b-aeae-48e2-a158-fb3e4fdff655\") " pod="opendatahub/maas-controller-6df954cb67-kr7xq"
Oct 30 20:16:23.263724 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.263554    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f638007b-aeae-48e2-a158-fb3e4fdff655-metrics-certs\") pod \"maas-controller-6df954cb67-kr7xq\" (UID: \"f638007b-aeae-48e2-a158-fb3e4fdff655\") " pod="opendatahub/maas-controller-6df954cb67-kr7xq"
Oct 30 20:16:23.263724 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.263629    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f638007b-aeae-48e2-a158-fb3e4fdff655-webhook-cert\") pod \"maas-controller-6df954cb67-kr7xq\" (UID: \"f638007b-aeae-48e2-a158-fb3e4fdff655\") " pod="opendatahub/maas-controller-6df954cb67-kr7xq"
Oct 30 20:16:23.364922 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.364891    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f638007b-aeae-48e2-a158-fb3e4fdff655-webhook-cert\") pod \"maas-controller-6df954cb67-kr7xq\" (UID: \"f638007b-aeae-48e2-a158-fb3e4fdff655\") " pod="opendatahub/maas-controller-6df954cb67-kr7xq"
Oct 30 20:16:23.365131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.364949    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5fv4\" (UniqueName: \"kubernetes.io/projected/f638007b-aeae-48e2-a158-fb3e4fdff655-kube-api-access-s5fv4\") pod \"maas-controller-6df954cb67-kr7xq\" (UID: \"f638007b-aeae-48e2-a158-fb3e4fdff655\") " pod="opendatahub/maas-controller-6df954cb67-kr7xq"
Oct 30 20:16:23.365131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.365005    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f638007b-aeae-48e2-a158-fb3e4fdff655-metrics-certs\") pod \"maas-controller-6df954cb67-kr7xq\" (UID: \"f638007b-aeae-48e2-a158-fb3e4fdff655\") " pod="opendatahub/maas-controller-6df954cb67-kr7xq"
Oct 30 20:16:23.368393 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.368344    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f638007b-aeae-48e2-a158-fb3e4fdff655-webhook-cert\") pod \"maas-controller-6df954cb67-kr7xq\" (UID: \"f638007b-aeae-48e2-a158-fb3e4fdff655\") " pod="opendatahub/maas-controller-6df954cb67-kr7xq"
Oct 30 20:16:23.369258 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.369229    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f638007b-aeae-48e2-a158-fb3e4fdff655-metrics-certs\") pod \"maas-controller-6df954cb67-kr7xq\" (UID: \"f638007b-aeae-48e2-a158-fb3e4fdff655\") " pod="opendatahub/maas-controller-6df954cb67-kr7xq"
Oct 30 20:16:23.378223 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.378197    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5fv4\" (UniqueName: \"kubernetes.io/projected/f638007b-aeae-48e2-a158-fb3e4fdff655-kube-api-access-s5fv4\") pod \"maas-controller-6df954cb67-kr7xq\" (UID: \"f638007b-aeae-48e2-a158-fb3e4fdff655\") " pod="opendatahub/maas-controller-6df954cb67-kr7xq"
Oct 30 20:16:23.515108 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.515080    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6df954cb67-kr7xq"
Oct 30 20:16:23.534877 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.534318    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e480f59-6b71-4efe-905f-b1959a4a3891" path="/var/lib/kubelet/pods/2e480f59-6b71-4efe-905f-b1959a4a3891/volumes"
Oct 30 20:16:23.787454 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.787433    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 20:16:23.850165 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.850069    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6df954cb67-kr7xq"]
Oct 30 20:16:23.868589 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.868331    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 20:16:23.873922 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:16:23.873900    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf638007b_aeae_48e2_a158_fb3e4fdff655.slice/crio-69f61f45aa82421dda015d0efa08367dd92219ca2b6fff74a977237bbb230385 WatchSource:0}: Error finding container 69f61f45aa82421dda015d0efa08367dd92219ca2b6fff74a977237bbb230385: Status 404 returned error can't find the container with id 69f61f45aa82421dda015d0efa08367dd92219ca2b6fff74a977237bbb230385
Oct 30 20:16:23.875268 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.875251    5408 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971188    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-credential-socket\") pod \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") "
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971231    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-workload-socket\") pod \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") "
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971264    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-credential-socket\") pod \"ad4e73d4-d56f-499f-ac82-01e963174b4d\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") "
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971292    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-token\" (UniqueName: \"kubernetes.io/projected/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-token\") pod \"ad4e73d4-d56f-499f-ac82-01e963174b4d\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") "
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971311    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-workload-socket\") pod \"ad4e73d4-d56f-499f-ac82-01e963174b4d\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") "
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971334    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-workload-certs\") pod \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") "
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971356    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4e73d4-d56f-499f-ac82-01e963174b4d-kube-api-access-gr498\" (UniqueName: \"kubernetes.io/projected/ad4e73d4-d56f-499f-ac82-01e963174b4d-kube-api-access-gr498\") pod \"ad4e73d4-d56f-499f-ac82-01e963174b4d\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") "
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971384    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-podinfo\") pod \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") "
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971407    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-data\") pod \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") "
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971436    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-token\" (UniqueName: \"kubernetes.io/projected/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-token\") pod \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") "
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971453    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-data\") pod \"ad4e73d4-d56f-499f-ac82-01e963174b4d\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") "
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971498    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-kube-api-access-5g5bm\" (UniqueName: \"kubernetes.io/projected/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-kube-api-access-5g5bm\") pod \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") "
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971520    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-envoy\") pod \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") "
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971553    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-workload-certs\") pod \"ad4e73d4-d56f-499f-ac82-01e963174b4d\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") "
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971565    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-credential-socket" pod "ad4e73d4-d56f-499f-ac82-01e963174b4d" (UID: "ad4e73d4-d56f-499f-ac82-01e963174b4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971579    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istiod-ca-cert\") pod \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\" (UID: \"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad\") "
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971644    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-podinfo\") pod \"ad4e73d4-d56f-499f-ac82-01e963174b4d\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") "
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971678    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad4e73d4-d56f-499f-ac82-01e963174b4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad4e73d4-d56f-499f-ac82-01e963174b4d-istiod-ca-cert\") pod \"ad4e73d4-d56f-499f-ac82-01e963174b4d\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") "
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971724    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-envoy\") pod \"ad4e73d4-d56f-499f-ac82-01e963174b4d\" (UID: \"ad4e73d4-d56f-499f-ac82-01e963174b4d\") "
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971881    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istiod-ca-cert" pod "4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" (UID: "4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.971934    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-workload-socket" pod "4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" (UID: "4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.972181    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.972205    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.972219    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.972235    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-data" pod "4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" (UID: "4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.972250    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-credential-socket" pod "4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" (UID: "4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.972412    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-workload-certs" pod "4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" (UID: "4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:23.972424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.972451    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-workload-socket" pod "ad4e73d4-d56f-499f-ac82-01e963174b4d" (UID: "ad4e73d4-d56f-499f-ac82-01e963174b4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:23.974667 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.972616    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-workload-certs" pod "ad4e73d4-d56f-499f-ac82-01e963174b4d" (UID: "ad4e73d4-d56f-499f-ac82-01e963174b4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:23.974667 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.972911    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-data" pod "ad4e73d4-d56f-499f-ac82-01e963174b4d" (UID: "ad4e73d4-d56f-499f-ac82-01e963174b4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:23.974667 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.972916    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4e73d4-d56f-499f-ac82-01e963174b4d-istiod-ca-cert" pod "ad4e73d4-d56f-499f-ac82-01e963174b4d" (UID: "ad4e73d4-d56f-499f-ac82-01e963174b4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:23.974967 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.974938    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-podinfo" pod "ad4e73d4-d56f-499f-ac82-01e963174b4d" (UID: "ad4e73d4-d56f-499f-ac82-01e963174b4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:16:23.975260 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.975183    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-token" pod "4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" (UID: "4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:23.975376 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.975253    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-kube-api-access-5g5bm" pod "4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" (UID: "4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad"). InnerVolumeSpecName "kube-api-access-5g5bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:23.975440 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.975377    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4e73d4-d56f-499f-ac82-01e963174b4d-kube-api-access-gr498" pod "ad4e73d4-d56f-499f-ac82-01e963174b4d" (UID: "ad4e73d4-d56f-499f-ac82-01e963174b4d"). InnerVolumeSpecName "kube-api-access-gr498". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:23.975440 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.975374    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-podinfo" pod "4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" (UID: "4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:16:23.975566 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.975502    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-token" pod "ad4e73d4-d56f-499f-ac82-01e963174b4d" (UID: "ad4e73d4-d56f-499f-ac82-01e963174b4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:23.976178 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.976160    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-envoy" pod "4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" (UID: "4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:23.976328 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:23.976308    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-envoy" pod "ad4e73d4-d56f-499f-ac82-01e963174b4d" (UID: "ad4e73d4-d56f-499f-ac82-01e963174b4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:24.073281 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.073210    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr498\" (UniqueName: \"kubernetes.io/projected/ad4e73d4-d56f-499f-ac82-01e963174b4d-kube-api-access-gr498\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.073281 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.073234    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.073281 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.073244    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.073281 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.073252    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.073281 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.073261    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.073281 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.073270    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g5bm\" (UniqueName: \"kubernetes.io/projected/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-kube-api-access-5g5bm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.073281 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.073278    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.073281 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.073286    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.073669 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.073294    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.073669 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.073302    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad4e73d4-d56f-499f-ac82-01e963174b4d-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.073669 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.073317    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.073669 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.073326    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.073669 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.073333    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad4e73d4-d56f-499f-ac82-01e963174b4d-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.073669 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.073341    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad4e73d4-d56f-499f-ac82-01e963174b4d-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.073669 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.073349    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.471647 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.471618    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6df954cb67-kr7xq" event={"ID":"f638007b-aeae-48e2-a158-fb3e4fdff655","Type":"ContainerStarted","Data":"69f61f45aa82421dda015d0efa08367dd92219ca2b6fff74a977237bbb230385"}
Oct 30 20:16:24.473008 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.472986    5408 generic.go:356] "Generic (PLEG): container finished" podID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" containerID="55539d73b315aa3a84ad5975871bbfde61277cd7de8088e65702be91617d8a57" exitCode=0
Oct 30 20:16:24.473126 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.473060    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" event={"ID":"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad","Type":"ContainerDied","Data":"55539d73b315aa3a84ad5975871bbfde61277cd7de8088e65702be91617d8a57"}
Oct 30 20:16:24.473126 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.473097    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr" event={"ID":"4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad","Type":"ContainerDied","Data":"d2d2045f1da198eaf5582c0e85545da77a1b3918d7ce6334a2bca9f407e0523e"}
Oct 30 20:16:24.473126 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.473118    5408 scope.go:122] "RemoveContainer" containerID="55539d73b315aa3a84ad5975871bbfde61277cd7de8088e65702be91617d8a57"
Oct 30 20:16:24.473380 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.473339    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"
Oct 30 20:16:24.474758 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.474737    5408 generic.go:356] "Generic (PLEG): container finished" podID="ad4e73d4-d56f-499f-ac82-01e963174b4d" containerID="5e9066a512fc0cac12e38031ad1e2f13d09b1cc6575a432eaaf83a282af39c11" exitCode=0
Oct 30 20:16:24.474870 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.474767    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs" event={"ID":"ad4e73d4-d56f-499f-ac82-01e963174b4d","Type":"ContainerDied","Data":"5e9066a512fc0cac12e38031ad1e2f13d09b1cc6575a432eaaf83a282af39c11"}
Oct 30 20:16:24.474870 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.474790    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"
Oct 30 20:16:24.474870 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.474796    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs" event={"ID":"ad4e73d4-d56f-499f-ac82-01e963174b4d","Type":"ContainerDied","Data":"f5e79c21c75164f90941a2a5a85baddbe7984861f55f38981bb5c9420e87c711"}
Oct 30 20:16:24.483873 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.483754    5408 scope.go:122] "RemoveContainer" containerID="55539d73b315aa3a84ad5975871bbfde61277cd7de8088e65702be91617d8a57"
Oct 30 20:16:24.484108 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:16:24.484081    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55539d73b315aa3a84ad5975871bbfde61277cd7de8088e65702be91617d8a57\": container with ID starting with 55539d73b315aa3a84ad5975871bbfde61277cd7de8088e65702be91617d8a57 not found: ID does not exist" containerID="55539d73b315aa3a84ad5975871bbfde61277cd7de8088e65702be91617d8a57"
Oct 30 20:16:24.484207 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.484120    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55539d73b315aa3a84ad5975871bbfde61277cd7de8088e65702be91617d8a57"} err="failed to get container status \"55539d73b315aa3a84ad5975871bbfde61277cd7de8088e65702be91617d8a57\": rpc error: code = NotFound desc = could not find container \"55539d73b315aa3a84ad5975871bbfde61277cd7de8088e65702be91617d8a57\": container with ID starting with 55539d73b315aa3a84ad5975871bbfde61277cd7de8088e65702be91617d8a57 not found: ID does not exist"
Oct 30 20:16:24.484207 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.484142    5408 scope.go:122] "RemoveContainer" containerID="5e9066a512fc0cac12e38031ad1e2f13d09b1cc6575a432eaaf83a282af39c11"
Oct 30 20:16:24.493172 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.493158    5408 scope.go:122] "RemoveContainer" containerID="5e9066a512fc0cac12e38031ad1e2f13d09b1cc6575a432eaaf83a282af39c11"
Oct 30 20:16:24.493432 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:16:24.493409    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e9066a512fc0cac12e38031ad1e2f13d09b1cc6575a432eaaf83a282af39c11\": container with ID starting with 5e9066a512fc0cac12e38031ad1e2f13d09b1cc6575a432eaaf83a282af39c11 not found: ID does not exist" containerID="5e9066a512fc0cac12e38031ad1e2f13d09b1cc6575a432eaaf83a282af39c11"
Oct 30 20:16:24.493477 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.493443    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9066a512fc0cac12e38031ad1e2f13d09b1cc6575a432eaaf83a282af39c11"} err="failed to get container status \"5e9066a512fc0cac12e38031ad1e2f13d09b1cc6575a432eaaf83a282af39c11\": rpc error: code = NotFound desc = could not find container \"5e9066a512fc0cac12e38031ad1e2f13d09b1cc6575a432eaaf83a282af39c11\": container with ID starting with 5e9066a512fc0cac12e38031ad1e2f13d09b1cc6575a432eaaf83a282af39c11 not found: ID does not exist"
Oct 30 20:16:24.503563 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.503527    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"]
Oct 30 20:16:24.512902 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.512057    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-98bzs"]
Oct 30 20:16:24.532398 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.531547    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"]
Oct 30 20:16:24.538893 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:24.538630    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-g86xr"]
Oct 30 20:16:25.482944 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:25.482910    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6df954cb67-kr7xq" event={"ID":"f638007b-aeae-48e2-a158-fb3e4fdff655","Type":"ContainerStarted","Data":"2d6d0694f18c8b660c7475eef3ae947e013ef7ee83c9b62b82180fb537842446"}
Oct 30 20:16:25.483133 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:25.483049    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6df954cb67-kr7xq"
Oct 30 20:16:25.517951 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:25.517889    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6df954cb67-kr7xq" podStartSLOduration=1.9987627 podStartE2EDuration="2.517871201s" totalImagesPullingTime="519.108501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:25.515166886 +0000 UTC m=+2740.099436536" watchObservedRunningTime="2026-10-30 20:16:25.517871201 +0000 UTC m=+2740.102140819"
Oct 30 20:16:25.542138 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:25.542111    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" path="/var/lib/kubelet/pods/4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad/volumes"
Oct 30 20:16:25.542571 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:25.542546    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad4e73d4-d56f-499f-ac82-01e963174b4d" path="/var/lib/kubelet/pods/ad4e73d4-d56f-499f-ac82-01e963174b4d/volumes"
Oct 30 20:16:36.489469 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:36.489436    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6df954cb67-kr7xq"
Oct 30 20:16:39.659666 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:16:39.659629    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:16:47.508380 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.508348    5408 scope.go:122] "RemoveContainer" containerID="3f16c1ffdfaecb09ebdd817ae433b9ef043340b16ecc6b54a96ff0fd1fcf0744"
Oct 30 20:16:47.516487 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.516468    5408 scope.go:122] "RemoveContainer" containerID="8ab2a93c1de41d19a4bf688736f37a059678061b040e21966340f9aad552e169"
Oct 30 20:16:47.711460 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.711422    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"]
Oct 30 20:16:47.711926 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.711905    5408 cpu_manager.go:414] "containerMap: removing container" podUID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" containerName="istio-proxy"
Oct 30 20:16:47.711926 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.711924    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" containerName="istio-proxy"
Oct 30 20:16:47.712098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.711936    5408 cpu_manager.go:414] "containerMap: removing container" podUID="ad4e73d4-d56f-499f-ac82-01e963174b4d" containerName="istio-proxy"
Oct 30 20:16:47.712098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.711940    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4e73d4-d56f-499f-ac82-01e963174b4d" containerName="istio-proxy"
Oct 30 20:16:47.712098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.712007    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae0c9d6-cbe9-43aa-8574-ba806c9f91ad" containerName="istio-proxy"
Oct 30 20:16:47.712098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.712014    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4e73d4-d56f-499f-ac82-01e963174b4d" containerName="istio-proxy"
Oct 30 20:16:47.716257 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.716239    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.735275 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.735246    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"]
Oct 30 20:16:47.742395 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.742371    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"]
Oct 30 20:16:47.746077 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.746061    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.785072 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.784977    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.785072 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.785018    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba388d10-05af-4652-b451-15f429ef1e4d-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.785386 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.785363    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.785486 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.785422    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.785553 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.785496    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2lvv\" (UniqueName: \"kubernetes.io/projected/ba388d10-05af-4652-b451-15f429ef1e4d-kube-api-access-d2lvv\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.785591 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.785567    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.785644 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.785595    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.785702 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.785686    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba388d10-05af-4652-b451-15f429ef1e4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.785748 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.785719    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba388d10-05af-4652-b451-15f429ef1e4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.786160 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.786137    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"]
Oct 30 20:16:47.886173 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886146    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba388d10-05af-4652-b451-15f429ef1e4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.886326 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886182    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6nz7\" (UniqueName: \"kubernetes.io/projected/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-kube-api-access-x6nz7\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.886326 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886221    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.886326 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886306    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.886512 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886348    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba388d10-05af-4652-b451-15f429ef1e4d-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.886512 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886390    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.886512 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886427    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.886686 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886526    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.886686 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886579    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.886686 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886613    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.886686 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886630    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2lvv\" (UniqueName: \"kubernetes.io/projected/ba388d10-05af-4652-b451-15f429ef1e4d-kube-api-access-d2lvv\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.886686 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886648    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.886686 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886669    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.886686 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886686    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.887179 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886709    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.887179 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886725    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.887179 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886747    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.887179 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886746    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.887179 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886768    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba388d10-05af-4652-b451-15f429ef1e4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.887179 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.886942    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.887179 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.887108    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.887525 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.887197    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.887525 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.887495    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba388d10-05af-4652-b451-15f429ef1e4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.889002 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.888977    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba388d10-05af-4652-b451-15f429ef1e4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.889268 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.889248    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.899447 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.899424    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba388d10-05af-4652-b451-15f429ef1e4d-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.900057 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.900020    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2lvv\" (UniqueName: \"kubernetes.io/projected/ba388d10-05af-4652-b451-15f429ef1e4d-kube-api-access-d2lvv\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cspst\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:47.987949 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.987923    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.988100 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.987970    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.988100 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.987992    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.988100 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.988016    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.988100 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.988051    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.988335 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.988177    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6nz7\" (UniqueName: \"kubernetes.io/projected/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-kube-api-access-x6nz7\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.988335 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.988222    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.988335 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.988264    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.988335 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.988299    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.988536 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.988384    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.988536 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.988441    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.988676 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.988655    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.988762 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.988738    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.988931 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.988915    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.990171 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.990152    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:47.990718 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:47.990703    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:48.000639 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:48.000617    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6nz7\" (UniqueName: \"kubernetes.io/projected/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-kube-api-access-x6nz7\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:48.000727 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:48.000678    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-dpg5s\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:48.030641 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:48.030618    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:48.060125 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:48.060096    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:48.312852 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:48.312771    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"]
Oct 30 20:16:48.315139 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:16:48.315116    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba388d10_05af_4652_b451_15f429ef1e4d.slice/crio-b33e1b222abc36d5ae23d15a963e9c753d707bbcc365856953ea346af72a0084 WatchSource:0}: Error finding container b33e1b222abc36d5ae23d15a963e9c753d707bbcc365856953ea346af72a0084: Status 404 returned error can't find the container with id b33e1b222abc36d5ae23d15a963e9c753d707bbcc365856953ea346af72a0084
Oct 30 20:16:48.317235 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:48.317203    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:16:48.317323 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:48.317264    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:16:48.317323 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:48.317293    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:16:48.347058 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:48.346402    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"]
Oct 30 20:16:48.372278 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:16:48.372252    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb32dd6_475e_4227_926e_f7eac2ae2ef7.slice/crio-687748a29377d3d48f7b09015da173d22616d9a3051632792bc62386d059902b WatchSource:0}: Error finding container 687748a29377d3d48f7b09015da173d22616d9a3051632792bc62386d059902b: Status 404 returned error can't find the container with id 687748a29377d3d48f7b09015da173d22616d9a3051632792bc62386d059902b
Oct 30 20:16:48.374553 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:48.374519    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:16:48.374661 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:48.374585    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:16:48.374661 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:48.374618    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:16:48.579371 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:48.579259    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s" event={"ID":"fbb32dd6-475e-4227-926e-f7eac2ae2ef7","Type":"ContainerStarted","Data":"6c2ddd07868e71d062ce663917fd377e6482cbf3937041efaf5e57af47373058"}
Oct 30 20:16:48.579371 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:48.579328    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s" event={"ID":"fbb32dd6-475e-4227-926e-f7eac2ae2ef7","Type":"ContainerStarted","Data":"687748a29377d3d48f7b09015da173d22616d9a3051632792bc62386d059902b"}
Oct 30 20:16:48.580805 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:48.580780    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst" event={"ID":"ba388d10-05af-4652-b451-15f429ef1e4d","Type":"ContainerStarted","Data":"30b057423f1fac89a5361a01d517db58ead21a2a58a14fdfcaab3cad483da4a2"}
Oct 30 20:16:48.580805 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:48.580825    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst" event={"ID":"ba388d10-05af-4652-b451-15f429ef1e4d","Type":"ContainerStarted","Data":"b33e1b222abc36d5ae23d15a963e9c753d707bbcc365856953ea346af72a0084"}
Oct 30 20:16:48.605393 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:48.605343    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s" podStartSLOduration=1.605325914 podStartE2EDuration="1.605325914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:48.602732673 +0000 UTC m=+2763.187002324" watchObservedRunningTime="2026-10-30 20:16:48.605325914 +0000 UTC m=+2763.189595531"
Oct 30 20:16:48.629724 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:48.629663    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst" podStartSLOduration=1.629645398 podStartE2EDuration="1.629645398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:48.627625941 +0000 UTC m=+2763.211895558" watchObservedRunningTime="2026-10-30 20:16:48.629645398 +0000 UTC m=+2763.213915016"
Oct 30 20:16:49.032878 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:49.032846    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:49.033250 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:49.033227    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cspst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Oct 30 20:16:49.033425 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:49.033400    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst" podUID="ba388d10-05af-4652-b451-15f429ef1e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Oct 30 20:16:49.060303 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:49.060272    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:49.061565 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:49.061536    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-dpg5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 20:16:49.061696 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:49.061609    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s" podUID="fbb32dd6-475e-4227-926e-f7eac2ae2ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 20:16:50.042054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:50.033401    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cspst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Oct 30 20:16:50.042054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:50.033464    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst" podUID="ba388d10-05af-4652-b451-15f429ef1e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Oct 30 20:16:50.061108 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:50.060807    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-dpg5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 20:16:50.061108 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:50.060855    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s" podUID="fbb32dd6-475e-4227-926e-f7eac2ae2ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 20:16:51.031466 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:51.031426    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cspst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Oct 30 20:16:51.031784 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:51.031755    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst" podUID="ba388d10-05af-4652-b451-15f429ef1e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Oct 30 20:16:51.061007 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:51.060980    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-dpg5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 20:16:51.061463 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:51.061441    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s" podUID="fbb32dd6-475e-4227-926e-f7eac2ae2ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 20:16:52.033063 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:52.032403    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cspst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Oct 30 20:16:52.033063 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:52.032466    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst" podUID="ba388d10-05af-4652-b451-15f429ef1e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Oct 30 20:16:52.060768 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:52.060730    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-dpg5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 20:16:52.060933 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:52.060781    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s" podUID="fbb32dd6-475e-4227-926e-f7eac2ae2ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 20:16:53.032053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:53.031424    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cspst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Oct 30 20:16:53.032053 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:53.031488    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst" podUID="ba388d10-05af-4652-b451-15f429ef1e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Oct 30 20:16:53.060615 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:53.060581    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-dpg5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 20:16:53.060771 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:53.060645    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s" podUID="fbb32dd6-475e-4227-926e-f7eac2ae2ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 20:16:54.043054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:54.034525    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cspst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Oct 30 20:16:54.043054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:54.034616    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst" podUID="ba388d10-05af-4652-b451-15f429ef1e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Oct 30 20:16:54.060808 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:54.060775    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-dpg5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 20:16:54.060961 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:54.060842    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s" podUID="fbb32dd6-475e-4227-926e-f7eac2ae2ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 20:16:54.545917 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:54.545877    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6df954cb67-kr7xq"]
Oct 30 20:16:54.546185 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:54.546140    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6df954cb67-kr7xq" podUID="f638007b-aeae-48e2-a158-fb3e4fdff655" containerName="manager" containerID="cri-o://2d6d0694f18c8b660c7475eef3ae947e013ef7ee83c9b62b82180fb537842446" gracePeriod=30
Oct 30 20:16:54.797127 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:54.797070    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6df954cb67-kr7xq"
Oct 30 20:16:54.901767 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:54.901732    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f638007b-aeae-48e2-a158-fb3e4fdff655-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f638007b-aeae-48e2-a158-fb3e4fdff655-webhook-cert\") pod \"f638007b-aeae-48e2-a158-fb3e4fdff655\" (UID: \"f638007b-aeae-48e2-a158-fb3e4fdff655\") "
Oct 30 20:16:54.901936 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:54.901789    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f638007b-aeae-48e2-a158-fb3e4fdff655-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f638007b-aeae-48e2-a158-fb3e4fdff655-metrics-certs\") pod \"f638007b-aeae-48e2-a158-fb3e4fdff655\" (UID: \"f638007b-aeae-48e2-a158-fb3e4fdff655\") "
Oct 30 20:16:54.901936 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:54.901855    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f638007b-aeae-48e2-a158-fb3e4fdff655-kube-api-access-s5fv4\" (UniqueName: \"kubernetes.io/projected/f638007b-aeae-48e2-a158-fb3e4fdff655-kube-api-access-s5fv4\") pod \"f638007b-aeae-48e2-a158-fb3e4fdff655\" (UID: \"f638007b-aeae-48e2-a158-fb3e4fdff655\") "
Oct 30 20:16:54.903999 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:54.903966    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f638007b-aeae-48e2-a158-fb3e4fdff655-webhook-cert" pod "f638007b-aeae-48e2-a158-fb3e4fdff655" (UID: "f638007b-aeae-48e2-a158-fb3e4fdff655"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:54.904422 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:54.904382    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f638007b-aeae-48e2-a158-fb3e4fdff655-metrics-certs" pod "f638007b-aeae-48e2-a158-fb3e4fdff655" (UID: "f638007b-aeae-48e2-a158-fb3e4fdff655"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:54.904520 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:54.904490    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f638007b-aeae-48e2-a158-fb3e4fdff655-kube-api-access-s5fv4" pod "f638007b-aeae-48e2-a158-fb3e4fdff655" (UID: "f638007b-aeae-48e2-a158-fb3e4fdff655"). InnerVolumeSpecName "kube-api-access-s5fv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:55.003398 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:55.003350    5408 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f638007b-aeae-48e2-a158-fb3e4fdff655-webhook-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:55.003398 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:55.003395    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f638007b-aeae-48e2-a158-fb3e4fdff655-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:55.003602 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:55.003411    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5fv4\" (UniqueName: \"kubernetes.io/projected/f638007b-aeae-48e2-a158-fb3e4fdff655-kube-api-access-s5fv4\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:55.031426 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:55.031394    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cspst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Oct 30 20:16:55.031548 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:55.031451    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst" podUID="ba388d10-05af-4652-b451-15f429ef1e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Oct 30 20:16:55.060948 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:55.060882    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-dpg5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 20:16:55.060948 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:55.060921    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s" podUID="fbb32dd6-475e-4227-926e-f7eac2ae2ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 20:16:55.612626 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:55.612586    5408 generic.go:356] "Generic (PLEG): container finished" podID="f638007b-aeae-48e2-a158-fb3e4fdff655" containerID="2d6d0694f18c8b660c7475eef3ae947e013ef7ee83c9b62b82180fb537842446" exitCode=0
Oct 30 20:16:55.612775 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:55.612644    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6df954cb67-kr7xq" event={"ID":"f638007b-aeae-48e2-a158-fb3e4fdff655","Type":"ContainerDied","Data":"2d6d0694f18c8b660c7475eef3ae947e013ef7ee83c9b62b82180fb537842446"}
Oct 30 20:16:55.612775 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:55.612653    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6df954cb67-kr7xq"
Oct 30 20:16:55.612775 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:55.612691    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6df954cb67-kr7xq" event={"ID":"f638007b-aeae-48e2-a158-fb3e4fdff655","Type":"ContainerDied","Data":"69f61f45aa82421dda015d0efa08367dd92219ca2b6fff74a977237bbb230385"}
Oct 30 20:16:55.612775 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:55.612713    5408 scope.go:122] "RemoveContainer" containerID="2d6d0694f18c8b660c7475eef3ae947e013ef7ee83c9b62b82180fb537842446"
Oct 30 20:16:55.621687 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:55.621664    5408 scope.go:122] "RemoveContainer" containerID="2d6d0694f18c8b660c7475eef3ae947e013ef7ee83c9b62b82180fb537842446"
Oct 30 20:16:55.621983 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:16:55.621959    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6d0694f18c8b660c7475eef3ae947e013ef7ee83c9b62b82180fb537842446\": container with ID starting with 2d6d0694f18c8b660c7475eef3ae947e013ef7ee83c9b62b82180fb537842446 not found: ID does not exist" containerID="2d6d0694f18c8b660c7475eef3ae947e013ef7ee83c9b62b82180fb537842446"
Oct 30 20:16:55.622138 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:55.621989    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6d0694f18c8b660c7475eef3ae947e013ef7ee83c9b62b82180fb537842446"} err="failed to get container status \"2d6d0694f18c8b660c7475eef3ae947e013ef7ee83c9b62b82180fb537842446\": rpc error: code = NotFound desc = could not find container \"2d6d0694f18c8b660c7475eef3ae947e013ef7ee83c9b62b82180fb537842446\": container with ID starting with 2d6d0694f18c8b660c7475eef3ae947e013ef7ee83c9b62b82180fb537842446 not found: ID does not exist"
Oct 30 20:16:55.632248 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:55.632222    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6df954cb67-kr7xq"]
Oct 30 20:16:55.637884 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:55.637864    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6df954cb67-kr7xq"]
Oct 30 20:16:56.592962 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:56.592923    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:56.594061 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:56.594022    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:56.594958 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:56.594938    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:16:56.630904 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:56.630870    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:56.631274 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:56.631257    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:56.631980 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:56.631961    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:16:56.920611 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:56.920530    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6df954cb67-w2ql2"]
Oct 30 20:16:56.920903 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:56.920889    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f638007b-aeae-48e2-a158-fb3e4fdff655" containerName="manager"
Oct 30 20:16:56.921001 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:56.920904    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f638007b-aeae-48e2-a158-fb3e4fdff655" containerName="manager"
Oct 30 20:16:56.921001 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:56.920979    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="f638007b-aeae-48e2-a158-fb3e4fdff655" containerName="manager"
Oct 30 20:16:56.925226 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:56.925208    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6df954cb67-w2ql2"
Oct 30 20:16:56.934386 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:56.934360    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 20:16:56.934552 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:56.934444    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2fq9m\""
Oct 30 20:16:56.934552 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:56.934522    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 20:16:56.934639 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:56.934584    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 20:16:56.935647 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:56.935622    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6df954cb67-w2ql2"]
Oct 30 20:16:57.023505 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:57.023463    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67ef2465-c1a0-4950-a13e-80170dcf1928-webhook-cert\") pod \"maas-controller-6df954cb67-w2ql2\" (UID: \"67ef2465-c1a0-4950-a13e-80170dcf1928\") " pod="opendatahub/maas-controller-6df954cb67-w2ql2"
Oct 30 20:16:57.023726 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:57.023605    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67ef2465-c1a0-4950-a13e-80170dcf1928-metrics-certs\") pod \"maas-controller-6df954cb67-w2ql2\" (UID: \"67ef2465-c1a0-4950-a13e-80170dcf1928\") " pod="opendatahub/maas-controller-6df954cb67-w2ql2"
Oct 30 20:16:57.023726 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:57.023700    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddb2t\" (UniqueName: \"kubernetes.io/projected/67ef2465-c1a0-4950-a13e-80170dcf1928-kube-api-access-ddb2t\") pod \"maas-controller-6df954cb67-w2ql2\" (UID: \"67ef2465-c1a0-4950-a13e-80170dcf1928\") " pod="opendatahub/maas-controller-6df954cb67-w2ql2"
Oct 30 20:16:57.124387 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:57.124345    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67ef2465-c1a0-4950-a13e-80170dcf1928-metrics-certs\") pod \"maas-controller-6df954cb67-w2ql2\" (UID: \"67ef2465-c1a0-4950-a13e-80170dcf1928\") " pod="opendatahub/maas-controller-6df954cb67-w2ql2"
Oct 30 20:16:57.124619 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:57.124407    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddb2t\" (UniqueName: \"kubernetes.io/projected/67ef2465-c1a0-4950-a13e-80170dcf1928-kube-api-access-ddb2t\") pod \"maas-controller-6df954cb67-w2ql2\" (UID: \"67ef2465-c1a0-4950-a13e-80170dcf1928\") " pod="opendatahub/maas-controller-6df954cb67-w2ql2"
Oct 30 20:16:57.124619 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:57.124438    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67ef2465-c1a0-4950-a13e-80170dcf1928-webhook-cert\") pod \"maas-controller-6df954cb67-w2ql2\" (UID: \"67ef2465-c1a0-4950-a13e-80170dcf1928\") " pod="opendatahub/maas-controller-6df954cb67-w2ql2"
Oct 30 20:16:57.126984 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:57.126957    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67ef2465-c1a0-4950-a13e-80170dcf1928-metrics-certs\") pod \"maas-controller-6df954cb67-w2ql2\" (UID: \"67ef2465-c1a0-4950-a13e-80170dcf1928\") " pod="opendatahub/maas-controller-6df954cb67-w2ql2"
Oct 30 20:16:57.126984 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:57.126971    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67ef2465-c1a0-4950-a13e-80170dcf1928-webhook-cert\") pod \"maas-controller-6df954cb67-w2ql2\" (UID: \"67ef2465-c1a0-4950-a13e-80170dcf1928\") " pod="opendatahub/maas-controller-6df954cb67-w2ql2"
Oct 30 20:16:57.155493 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:57.155459    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddb2t\" (UniqueName: \"kubernetes.io/projected/67ef2465-c1a0-4950-a13e-80170dcf1928-kube-api-access-ddb2t\") pod \"maas-controller-6df954cb67-w2ql2\" (UID: \"67ef2465-c1a0-4950-a13e-80170dcf1928\") " pod="opendatahub/maas-controller-6df954cb67-w2ql2"
Oct 30 20:16:57.238602 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:57.238567    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6df954cb67-w2ql2"
Oct 30 20:16:57.529858 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:57.529777    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f638007b-aeae-48e2-a158-fb3e4fdff655" path="/var/lib/kubelet/pods/f638007b-aeae-48e2-a158-fb3e4fdff655/volumes"
Oct 30 20:16:57.606314 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:57.606283    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6df954cb67-w2ql2"]
Oct 30 20:16:57.611957 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:16:57.611930    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ef2465_c1a0_4950_a13e_80170dcf1928.slice/crio-1bb3035b44856d78b9297a859a06b244295333355df18bea85b107ff083b3125 WatchSource:0}: Error finding container 1bb3035b44856d78b9297a859a06b244295333355df18bea85b107ff083b3125: Status 404 returned error can't find the container with id 1bb3035b44856d78b9297a859a06b244295333355df18bea85b107ff083b3125
Oct 30 20:16:57.623482 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:57.623457    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6df954cb67-w2ql2" event={"ID":"67ef2465-c1a0-4950-a13e-80170dcf1928","Type":"ContainerStarted","Data":"1bb3035b44856d78b9297a859a06b244295333355df18bea85b107ff083b3125"}
Oct 30 20:16:58.629342 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:58.629299    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6df954cb67-w2ql2" event={"ID":"67ef2465-c1a0-4950-a13e-80170dcf1928","Type":"ContainerStarted","Data":"229c6c7cc32a6a48b8c19837d071a2bb2239a483a7d06fe34d6e774cb562bb62"}
Oct 30 20:16:58.629755 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:58.629647    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6df954cb67-w2ql2"
Oct 30 20:16:58.651982 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:16:58.651926    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6df954cb67-w2ql2" podStartSLOduration=2.141694381 podStartE2EDuration="2.651909622s" totalImagesPullingTime="510.215241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:58.650854798 +0000 UTC m=+2773.235124416" watchObservedRunningTime="2026-10-30 20:16:58.651909622 +0000 UTC m=+2773.236179240"
Oct 30 20:17:09.636123 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:17:09.636096    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6df954cb67-w2ql2"
Oct 30 20:17:09.660482 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:17:09.660444    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:17:39.659944 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:17:39.659864    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:18:09.659246 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:18:09.659209    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:18:27.636178 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:18:27.636135    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cspst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:18:27.636946 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:18:27.636911    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst" podUID="ba388d10-05af-4652-b451-15f429ef1e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:18:27.679180 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:18:27.679146    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-dpg5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:18:27.679499 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:18:27.679469    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s" podUID="fbb32dd6-475e-4227-926e-f7eac2ae2ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:18:39.659420 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:18:39.659383    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:18:52.429200 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:18:52.429155    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-2wd98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:18:52.429947 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:18:52.429909    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98" podUID="adfef3ed-77c2-4817-9a2c-96403e22f83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:04.489112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:19:04.488183    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-h68ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:19:04.489112 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:19:04.488294    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls" podUID="e6ba43a0-ddfc-459d-8e82-1c2e197557a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:19:09.658385 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:19:09.658347    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:19:39.660139 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:19:39.660101    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:20:09.659692 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:20:09.659657    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:20:39.659348 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:20:39.659268    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:21:09.658476 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:21:09.658441    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:21:39.658751 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:21:39.658708    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:22:09.662385 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:22:09.662300    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:22:39.659454 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:22:39.659419    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:23:09.659190 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:23:09.659153    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:23:39.658921 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:23:39.658836    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:24:09.659629 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:24:09.659589    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:24:39.659394 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:24:39.659356    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:25:09.659537 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:25:09.659458    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:25:39.659426 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:25:39.659316    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:26:09.660161 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:26:09.660121    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:26:39.659611 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:26:39.659521    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:26:48.486055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:48.479462    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"]
Oct 30 20:26:48.486055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:48.479750    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst" podUID="ba388d10-05af-4652-b451-15f429ef1e4d" containerName="istio-proxy" containerID="cri-o://30b057423f1fac89a5361a01d517db58ead21a2a58a14fdfcaab3cad483da4a2" gracePeriod=30
Oct 30 20:26:48.500143 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:48.500115    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"]
Oct 30 20:26:48.500366 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:48.500350    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"]
Oct 30 20:26:48.500667 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:48.500640    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98" podUID="adfef3ed-77c2-4817-9a2c-96403e22f83b" containerName="istio-proxy" containerID="cri-o://b31d2af6fc155b5e0f2807f21faf641f3e6a09819c93ae77d88fa83ec9557df4" gracePeriod=30
Oct 30 20:26:48.501106 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:48.501079    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s" podUID="fbb32dd6-475e-4227-926e-f7eac2ae2ef7" containerName="istio-proxy" containerID="cri-o://6c2ddd07868e71d062ce663917fd377e6482cbf3937041efaf5e57af47373058" gracePeriod=30
Oct 30 20:26:48.502728 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:48.502691    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"]
Oct 30 20:26:48.544328 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:48.544258    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls" podUID="e6ba43a0-ddfc-459d-8e82-1c2e197557a0" containerName="istio-proxy" containerID="cri-o://7b6a44087634fbb74888a6a62d5b26b76705c9b10cf5b2e0617fd8b0809bef8b" gracePeriod=30
Oct 30 20:26:49.491295 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:49.491140    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-h68ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:26:49.491295 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:49.491204    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls" podUID="e6ba43a0-ddfc-459d-8e82-1c2e197557a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:26:51.385767 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:51.385734    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-2wd98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:26:51.386315 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:51.385793    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98" podUID="adfef3ed-77c2-4817-9a2c-96403e22f83b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:26:54.704423 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.704394    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 20:26:54.884701 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.884664    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-token\" (UniqueName: \"kubernetes.io/projected/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-token\") pod \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") "
Oct 30 20:26:54.884864 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.884723    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-envoy\") pod \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") "
Oct 30 20:26:54.884864 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.884778    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-podinfo\") pod \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") "
Oct 30 20:26:54.884864 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.884824    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-workload-socket\") pod \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") "
Oct 30 20:26:54.884864 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.884858    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-credential-socket\") pod \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") "
Oct 30 20:26:54.885138 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.884917    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-workload-certs\") pod \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") "
Oct 30 20:26:54.885138 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.884946    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istiod-ca-cert\") pod \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") "
Oct 30 20:26:54.885138 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.884984    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-kube-api-access-mxhf9\" (UniqueName: \"kubernetes.io/projected/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-kube-api-access-mxhf9\") pod \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") "
Oct 30 20:26:54.888131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.886149    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-credential-socket" pod "e6ba43a0-ddfc-459d-8e82-1c2e197557a0" (UID: "e6ba43a0-ddfc-459d-8e82-1c2e197557a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.888131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.886363    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-workload-certs" pod "e6ba43a0-ddfc-459d-8e82-1c2e197557a0" (UID: "e6ba43a0-ddfc-459d-8e82-1c2e197557a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.888131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.886274    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istiod-ca-cert" pod "e6ba43a0-ddfc-459d-8e82-1c2e197557a0" (UID: "e6ba43a0-ddfc-459d-8e82-1c2e197557a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:54.888131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.886539    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-data\") pod \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\" (UID: \"e6ba43a0-ddfc-459d-8e82-1c2e197557a0\") "
Oct 30 20:26:54.888131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.886567    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-workload-socket" pod "e6ba43a0-ddfc-459d-8e82-1c2e197557a0" (UID: "e6ba43a0-ddfc-459d-8e82-1c2e197557a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.888131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.886856    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-data" pod "e6ba43a0-ddfc-459d-8e82-1c2e197557a0" (UID: "e6ba43a0-ddfc-459d-8e82-1c2e197557a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.888131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.887007    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.888131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.887045    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.888131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.887061    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.888131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.887076    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.888131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.887090    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.888131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.888081    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-envoy" pod "e6ba43a0-ddfc-459d-8e82-1c2e197557a0" (UID: "e6ba43a0-ddfc-459d-8e82-1c2e197557a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.888858 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.888842    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-podinfo" pod "e6ba43a0-ddfc-459d-8e82-1c2e197557a0" (UID: "e6ba43a0-ddfc-459d-8e82-1c2e197557a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:26:54.888915 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.888877    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-kube-api-access-mxhf9" pod "e6ba43a0-ddfc-459d-8e82-1c2e197557a0" (UID: "e6ba43a0-ddfc-459d-8e82-1c2e197557a0"). InnerVolumeSpecName "kube-api-access-mxhf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:54.888954 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.888916    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-token" pod "e6ba43a0-ddfc-459d-8e82-1c2e197557a0" (UID: "e6ba43a0-ddfc-459d-8e82-1c2e197557a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:54.893155 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.893135    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:26:54.987953 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.987918    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-istio-data\") pod \"ba388d10-05af-4652-b451-15f429ef1e4d\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") "
Oct 30 20:26:54.988135 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.987965    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba388d10-05af-4652-b451-15f429ef1e4d-kube-api-access-d2lvv\" (UniqueName: \"kubernetes.io/projected/ba388d10-05af-4652-b451-15f429ef1e4d-kube-api-access-d2lvv\") pod \"ba388d10-05af-4652-b451-15f429ef1e4d\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") "
Oct 30 20:26:54.988135 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988016    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-workload-certs\") pod \"ba388d10-05af-4652-b451-15f429ef1e4d\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") "
Oct 30 20:26:54.988135 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988087    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-istio-envoy\") pod \"ba388d10-05af-4652-b451-15f429ef1e4d\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") "
Oct 30 20:26:54.988135 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988117    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba388d10-05af-4652-b451-15f429ef1e4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba388d10-05af-4652-b451-15f429ef1e4d-istio-podinfo\") pod \"ba388d10-05af-4652-b451-15f429ef1e4d\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") "
Oct 30 20:26:54.988135 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988138    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba388d10-05af-4652-b451-15f429ef1e4d-istio-token\" (UniqueName: \"kubernetes.io/projected/ba388d10-05af-4652-b451-15f429ef1e4d-istio-token\") pod \"ba388d10-05af-4652-b451-15f429ef1e4d\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") "
Oct 30 20:26:54.988501 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988167    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba388d10-05af-4652-b451-15f429ef1e4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba388d10-05af-4652-b451-15f429ef1e4d-istiod-ca-cert\") pod \"ba388d10-05af-4652-b451-15f429ef1e4d\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") "
Oct 30 20:26:54.988501 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988200    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-credential-socket\") pod \"ba388d10-05af-4652-b451-15f429ef1e4d\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") "
Oct 30 20:26:54.988501 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988224    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-istio-data" pod "ba388d10-05af-4652-b451-15f429ef1e4d" (UID: "ba388d10-05af-4652-b451-15f429ef1e4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.988501 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988231    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-workload-socket\") pod \"ba388d10-05af-4652-b451-15f429ef1e4d\" (UID: \"ba388d10-05af-4652-b451-15f429ef1e4d\") "
Oct 30 20:26:54.988501 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988444    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-workload-certs" pod "ba388d10-05af-4652-b451-15f429ef1e4d" (UID: "ba388d10-05af-4652-b451-15f429ef1e4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.988501 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988455    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-workload-socket" pod "ba388d10-05af-4652-b451-15f429ef1e4d" (UID: "ba388d10-05af-4652-b451-15f429ef1e4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.988836 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988627    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-credential-socket" pod "ba388d10-05af-4652-b451-15f429ef1e4d" (UID: "ba388d10-05af-4652-b451-15f429ef1e4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.988836 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988653    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.988836 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988671    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.988836 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988684    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.988836 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988698    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.988836 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988713    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.988836 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988726    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxhf9\" (UniqueName: \"kubernetes.io/projected/e6ba43a0-ddfc-459d-8e82-1c2e197557a0-kube-api-access-mxhf9\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.988836 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988737    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.988836 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.988736    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba388d10-05af-4652-b451-15f429ef1e4d-istiod-ca-cert" pod "ba388d10-05af-4652-b451-15f429ef1e4d" (UID: "ba388d10-05af-4652-b451-15f429ef1e4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:54.990352 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.990329    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba388d10-05af-4652-b451-15f429ef1e4d-istio-podinfo" pod "ba388d10-05af-4652-b451-15f429ef1e4d" (UID: "ba388d10-05af-4652-b451-15f429ef1e4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:26:54.990471 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.990455    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-istio-envoy" pod "ba388d10-05af-4652-b451-15f429ef1e4d" (UID: "ba388d10-05af-4652-b451-15f429ef1e4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.990786 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.990766    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba388d10-05af-4652-b451-15f429ef1e4d-istio-token" pod "ba388d10-05af-4652-b451-15f429ef1e4d" (UID: "ba388d10-05af-4652-b451-15f429ef1e4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:54.990863 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:54.990799    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba388d10-05af-4652-b451-15f429ef1e4d-kube-api-access-d2lvv" pod "ba388d10-05af-4652-b451-15f429ef1e4d" (UID: "ba388d10-05af-4652-b451-15f429ef1e4d"). InnerVolumeSpecName "kube-api-access-d2lvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:55.089395 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.089329    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2lvv\" (UniqueName: \"kubernetes.io/projected/ba388d10-05af-4652-b451-15f429ef1e4d-kube-api-access-d2lvv\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.089395 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.089353    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.089395 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.089364    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba388d10-05af-4652-b451-15f429ef1e4d-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.089395 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.089373    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba388d10-05af-4652-b451-15f429ef1e4d-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.089395 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.089381    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba388d10-05af-4652-b451-15f429ef1e4d-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.089395 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.089389    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba388d10-05af-4652-b451-15f429ef1e4d-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.405105 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.405060    5408 generic.go:356] "Generic (PLEG): container finished" podID="ba388d10-05af-4652-b451-15f429ef1e4d" containerID="30b057423f1fac89a5361a01d517db58ead21a2a58a14fdfcaab3cad483da4a2" exitCode=0
Oct 30 20:26:55.405105 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.405104    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"
Oct 30 20:26:55.405357 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.405140    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst" event={"ID":"ba388d10-05af-4652-b451-15f429ef1e4d","Type":"ContainerDied","Data":"30b057423f1fac89a5361a01d517db58ead21a2a58a14fdfcaab3cad483da4a2"}
Oct 30 20:26:55.405357 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.405207    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst" event={"ID":"ba388d10-05af-4652-b451-15f429ef1e4d","Type":"ContainerDied","Data":"b33e1b222abc36d5ae23d15a963e9c753d707bbcc365856953ea346af72a0084"}
Oct 30 20:26:55.405357 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.405240    5408 scope.go:122] "RemoveContainer" containerID="30b057423f1fac89a5361a01d517db58ead21a2a58a14fdfcaab3cad483da4a2"
Oct 30 20:26:55.407074 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.407049    5408 generic.go:356] "Generic (PLEG): container finished" podID="fbb32dd6-475e-4227-926e-f7eac2ae2ef7" containerID="6c2ddd07868e71d062ce663917fd377e6482cbf3937041efaf5e57af47373058" exitCode=0
Oct 30 20:26:55.407209 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.407137    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s" event={"ID":"fbb32dd6-475e-4227-926e-f7eac2ae2ef7","Type":"ContainerDied","Data":"6c2ddd07868e71d062ce663917fd377e6482cbf3937041efaf5e57af47373058"}
Oct 30 20:26:55.408991 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.408967    5408 generic.go:356] "Generic (PLEG): container finished" podID="adfef3ed-77c2-4817-9a2c-96403e22f83b" containerID="b31d2af6fc155b5e0f2807f21faf641f3e6a09819c93ae77d88fa83ec9557df4" exitCode=0
Oct 30 20:26:55.409105 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.409007    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98" event={"ID":"adfef3ed-77c2-4817-9a2c-96403e22f83b","Type":"ContainerDied","Data":"b31d2af6fc155b5e0f2807f21faf641f3e6a09819c93ae77d88fa83ec9557df4"}
Oct 30 20:26:55.410676 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.410652    5408 generic.go:356] "Generic (PLEG): container finished" podID="e6ba43a0-ddfc-459d-8e82-1c2e197557a0" containerID="7b6a44087634fbb74888a6a62d5b26b76705c9b10cf5b2e0617fd8b0809bef8b" exitCode=0
Oct 30 20:26:55.410773 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.410726    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls" event={"ID":"e6ba43a0-ddfc-459d-8e82-1c2e197557a0","Type":"ContainerDied","Data":"7b6a44087634fbb74888a6a62d5b26b76705c9b10cf5b2e0617fd8b0809bef8b"}
Oct 30 20:26:55.410773 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.410766    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls" event={"ID":"e6ba43a0-ddfc-459d-8e82-1c2e197557a0","Type":"ContainerDied","Data":"57aba8b251da3d47d5e5d1bdd81f82eea5a4faf43ea6be22dd1367e7e6dec58f"}
Oct 30 20:26:55.410912 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.410791    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"
Oct 30 20:26:55.415443 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.415426    5408 scope.go:122] "RemoveContainer" containerID="30b057423f1fac89a5361a01d517db58ead21a2a58a14fdfcaab3cad483da4a2"
Oct 30 20:26:55.415751 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:26:55.415725    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30b057423f1fac89a5361a01d517db58ead21a2a58a14fdfcaab3cad483da4a2\": container with ID starting with 30b057423f1fac89a5361a01d517db58ead21a2a58a14fdfcaab3cad483da4a2 not found: ID does not exist" containerID="30b057423f1fac89a5361a01d517db58ead21a2a58a14fdfcaab3cad483da4a2"
Oct 30 20:26:55.415832 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.415763    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30b057423f1fac89a5361a01d517db58ead21a2a58a14fdfcaab3cad483da4a2"} err="failed to get container status \"30b057423f1fac89a5361a01d517db58ead21a2a58a14fdfcaab3cad483da4a2\": rpc error: code = NotFound desc = could not find container \"30b057423f1fac89a5361a01d517db58ead21a2a58a14fdfcaab3cad483da4a2\": container with ID starting with 30b057423f1fac89a5361a01d517db58ead21a2a58a14fdfcaab3cad483da4a2 not found: ID does not exist"
Oct 30 20:26:55.415832 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.415786    5408 scope.go:122] "RemoveContainer" containerID="7b6a44087634fbb74888a6a62d5b26b76705c9b10cf5b2e0617fd8b0809bef8b"
Oct 30 20:26:55.425565 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.425507    5408 scope.go:122] "RemoveContainer" containerID="7b6a44087634fbb74888a6a62d5b26b76705c9b10cf5b2e0617fd8b0809bef8b"
Oct 30 20:26:55.425820 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:26:55.425793    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6a44087634fbb74888a6a62d5b26b76705c9b10cf5b2e0617fd8b0809bef8b\": container with ID starting with 7b6a44087634fbb74888a6a62d5b26b76705c9b10cf5b2e0617fd8b0809bef8b not found: ID does not exist" containerID="7b6a44087634fbb74888a6a62d5b26b76705c9b10cf5b2e0617fd8b0809bef8b"
Oct 30 20:26:55.425872 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.425832    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6a44087634fbb74888a6a62d5b26b76705c9b10cf5b2e0617fd8b0809bef8b"} err="failed to get container status \"7b6a44087634fbb74888a6a62d5b26b76705c9b10cf5b2e0617fd8b0809bef8b\": rpc error: code = NotFound desc = could not find container \"7b6a44087634fbb74888a6a62d5b26b76705c9b10cf5b2e0617fd8b0809bef8b\": container with ID starting with 7b6a44087634fbb74888a6a62d5b26b76705c9b10cf5b2e0617fd8b0809bef8b not found: ID does not exist"
Oct 30 20:26:55.431115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.431097    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 20:26:55.432974 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.432957    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:26:55.434580 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.434561    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"]
Oct 30 20:26:55.439385 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.439365    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cspst"]
Oct 30 20:26:55.478776 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.478752    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"]
Oct 30 20:26:55.484519 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.484485    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-h68ls"]
Oct 30 20:26:55.529046 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.529012    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba388d10-05af-4652-b451-15f429ef1e4d" path="/var/lib/kubelet/pods/ba388d10-05af-4652-b451-15f429ef1e4d/volumes"
Oct 30 20:26:55.529528 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.529512    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ba43a0-ddfc-459d-8e82-1c2e197557a0" path="/var/lib/kubelet/pods/e6ba43a0-ddfc-459d-8e82-1c2e197557a0/volumes"
Oct 30 20:26:55.594099 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594068    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-workload-certs\") pod \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") "
Oct 30 20:26:55.594313 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594120    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adfef3ed-77c2-4817-9a2c-96403e22f83b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adfef3ed-77c2-4817-9a2c-96403e22f83b-istiod-ca-cert\") pod \"adfef3ed-77c2-4817-9a2c-96403e22f83b\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") "
Oct 30 20:26:55.594313 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594160    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-token\" (UniqueName: \"kubernetes.io/projected/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-token\") pod \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") "
Oct 30 20:26:55.594313 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594187    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adfef3ed-77c2-4817-9a2c-96403e22f83b-kube-api-access-7wxg7\" (UniqueName: \"kubernetes.io/projected/adfef3ed-77c2-4817-9a2c-96403e22f83b-kube-api-access-7wxg7\") pod \"adfef3ed-77c2-4817-9a2c-96403e22f83b\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") "
Oct 30 20:26:55.594313 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594220    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-workload-socket\") pod \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") "
Oct 30 20:26:55.594313 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594255    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-workload-certs\") pod \"adfef3ed-77c2-4817-9a2c-96403e22f83b\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") "
Oct 30 20:26:55.594313 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594291    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-credential-socket\") pod \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") "
Oct 30 20:26:55.594656 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594361    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-envoy\") pod \"adfef3ed-77c2-4817-9a2c-96403e22f83b\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") "
Oct 30 20:26:55.594656 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594388    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istiod-ca-cert\") pod \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") "
Oct 30 20:26:55.594656 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594415    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-workload-certs" pod "fbb32dd6-475e-4227-926e-f7eac2ae2ef7" (UID: "fbb32dd6-475e-4227-926e-f7eac2ae2ef7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:55.594656 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594444    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-kube-api-access-x6nz7\" (UniqueName: \"kubernetes.io/projected/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-kube-api-access-x6nz7\") pod \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") "
Oct 30 20:26:55.594656 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594606    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-data\") pod \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") "
Oct 30 20:26:55.594656 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594651    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-token\" (UniqueName: \"kubernetes.io/projected/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-token\") pod \"adfef3ed-77c2-4817-9a2c-96403e22f83b\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") "
Oct 30 20:26:55.594971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594624    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-workload-certs" pod "adfef3ed-77c2-4817-9a2c-96403e22f83b" (UID: "adfef3ed-77c2-4817-9a2c-96403e22f83b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:55.594971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594685    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfef3ed-77c2-4817-9a2c-96403e22f83b-istiod-ca-cert" pod "adfef3ed-77c2-4817-9a2c-96403e22f83b" (UID: "adfef3ed-77c2-4817-9a2c-96403e22f83b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:55.594971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594691    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-credential-socket\") pod \"adfef3ed-77c2-4817-9a2c-96403e22f83b\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") "
Oct 30 20:26:55.594971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594702    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-workload-socket" pod "fbb32dd6-475e-4227-926e-f7eac2ae2ef7" (UID: "fbb32dd6-475e-4227-926e-f7eac2ae2ef7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:55.594971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594751    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-podinfo\") pod \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") "
Oct 30 20:26:55.594971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594790    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-data\") pod \"adfef3ed-77c2-4817-9a2c-96403e22f83b\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") "
Oct 30 20:26:55.594971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594817    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-podinfo\") pod \"adfef3ed-77c2-4817-9a2c-96403e22f83b\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") "
Oct 30 20:26:55.594971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594864    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-credential-socket" pod "fbb32dd6-475e-4227-926e-f7eac2ae2ef7" (UID: "fbb32dd6-475e-4227-926e-f7eac2ae2ef7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:55.594971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594948    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-workload-socket\") pod \"adfef3ed-77c2-4817-9a2c-96403e22f83b\" (UID: \"adfef3ed-77c2-4817-9a2c-96403e22f83b\") "
Oct 30 20:26:55.595586 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594958    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-credential-socket" pod "adfef3ed-77c2-4817-9a2c-96403e22f83b" (UID: "adfef3ed-77c2-4817-9a2c-96403e22f83b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:55.595586 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.594961    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-data" pod "fbb32dd6-475e-4227-926e-f7eac2ae2ef7" (UID: "fbb32dd6-475e-4227-926e-f7eac2ae2ef7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:55.595586 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.595234    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-workload-socket" pod "adfef3ed-77c2-4817-9a2c-96403e22f83b" (UID: "adfef3ed-77c2-4817-9a2c-96403e22f83b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:55.595751 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.595630    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istiod-ca-cert" pod "fbb32dd6-475e-4227-926e-f7eac2ae2ef7" (UID: "fbb32dd6-475e-4227-926e-f7eac2ae2ef7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:55.595898 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.595874    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-data" pod "adfef3ed-77c2-4817-9a2c-96403e22f83b" (UID: "adfef3ed-77c2-4817-9a2c-96403e22f83b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:55.595962 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.595922    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-envoy\") pod \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\" (UID: \"fbb32dd6-475e-4227-926e-f7eac2ae2ef7\") "
Oct 30 20:26:55.596254 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.596233    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.596366 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.596258    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.596366 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.596273    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.596366 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.596286    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.596366 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.596299    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.596366 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.596314    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.596366 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.596328    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adfef3ed-77c2-4817-9a2c-96403e22f83b-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.596366 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.596342    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.596366 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.596358    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.596804 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.596372    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.597676 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.597649    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-podinfo" pod "fbb32dd6-475e-4227-926e-f7eac2ae2ef7" (UID: "fbb32dd6-475e-4227-926e-f7eac2ae2ef7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:26:55.597676 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.597667    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-podinfo" pod "adfef3ed-77c2-4817-9a2c-96403e22f83b" (UID: "adfef3ed-77c2-4817-9a2c-96403e22f83b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:26:55.597974 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.597948    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-envoy" pod "adfef3ed-77c2-4817-9a2c-96403e22f83b" (UID: "adfef3ed-77c2-4817-9a2c-96403e22f83b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:55.598103 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.598082    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-token" pod "fbb32dd6-475e-4227-926e-f7eac2ae2ef7" (UID: "fbb32dd6-475e-4227-926e-f7eac2ae2ef7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:55.598152 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.598124    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-token" pod "adfef3ed-77c2-4817-9a2c-96403e22f83b" (UID: "adfef3ed-77c2-4817-9a2c-96403e22f83b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:55.598152 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.598144    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-kube-api-access-x6nz7" pod "fbb32dd6-475e-4227-926e-f7eac2ae2ef7" (UID: "fbb32dd6-475e-4227-926e-f7eac2ae2ef7"). InnerVolumeSpecName "kube-api-access-x6nz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:55.598745 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.598722    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-envoy" pod "fbb32dd6-475e-4227-926e-f7eac2ae2ef7" (UID: "fbb32dd6-475e-4227-926e-f7eac2ae2ef7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:55.598832 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.598743    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adfef3ed-77c2-4817-9a2c-96403e22f83b-kube-api-access-7wxg7" pod "adfef3ed-77c2-4817-9a2c-96403e22f83b" (UID: "adfef3ed-77c2-4817-9a2c-96403e22f83b"). InnerVolumeSpecName "kube-api-access-7wxg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:55.697717 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.697686    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.697717 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.697710    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6nz7\" (UniqueName: \"kubernetes.io/projected/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-kube-api-access-x6nz7\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.697717 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.697721    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.697940 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.697733    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.697940 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.697746    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adfef3ed-77c2-4817-9a2c-96403e22f83b-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.697940 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.697757    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.697940 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.697767    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb32dd6-475e-4227-926e-f7eac2ae2ef7-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.697940 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:55.697775    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wxg7\" (UniqueName: \"kubernetes.io/projected/adfef3ed-77c2-4817-9a2c-96403e22f83b-kube-api-access-7wxg7\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:56.417578 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:56.417548    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"
Oct 30 20:26:56.418068 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:56.417547    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s" event={"ID":"fbb32dd6-475e-4227-926e-f7eac2ae2ef7","Type":"ContainerDied","Data":"687748a29377d3d48f7b09015da173d22616d9a3051632792bc62386d059902b"}
Oct 30 20:26:56.418068 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:56.417701    5408 scope.go:122] "RemoveContainer" containerID="6c2ddd07868e71d062ce663917fd377e6482cbf3937041efaf5e57af47373058"
Oct 30 20:26:56.419010 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:56.418916    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98" event={"ID":"adfef3ed-77c2-4817-9a2c-96403e22f83b","Type":"ContainerDied","Data":"65e490e58bddbc7cd4399ae29f5ba712db5b472df7bbb073ab9e4272c224dc0b"}
Oct 30 20:26:56.419010 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:56.418926    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"
Oct 30 20:26:56.427723 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:56.427707    5408 scope.go:122] "RemoveContainer" containerID="b31d2af6fc155b5e0f2807f21faf641f3e6a09819c93ae77d88fa83ec9557df4"
Oct 30 20:26:56.463081 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:56.463061    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"]
Oct 30 20:26:56.476098 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:56.476076    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2wd98"]
Oct 30 20:26:56.494764 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:56.494746    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"]
Oct 30 20:26:56.512581 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:56.512564    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-dpg5s"]
Oct 30 20:26:57.528884 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:57.528853    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adfef3ed-77c2-4817-9a2c-96403e22f83b" path="/var/lib/kubelet/pods/adfef3ed-77c2-4817-9a2c-96403e22f83b/volumes"
Oct 30 20:26:57.529383 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:26:57.529303    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb32dd6-475e-4227-926e-f7eac2ae2ef7" path="/var/lib/kubelet/pods/fbb32dd6-475e-4227-926e-f7eac2ae2ef7/volumes"
Oct 30 20:27:09.659523 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:27:09.659489    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:27:18.565264 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.565220    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"]
Oct 30 20:27:18.565788 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.565680    5408 cpu_manager.go:414] "containerMap: removing container" podUID="e6ba43a0-ddfc-459d-8e82-1c2e197557a0" containerName="istio-proxy"
Oct 30 20:27:18.565788 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.565697    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ba43a0-ddfc-459d-8e82-1c2e197557a0" containerName="istio-proxy"
Oct 30 20:27:18.565788 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.565712    5408 cpu_manager.go:414] "containerMap: removing container" podUID="ba388d10-05af-4652-b451-15f429ef1e4d" containerName="istio-proxy"
Oct 30 20:27:18.565788 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.565719    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba388d10-05af-4652-b451-15f429ef1e4d" containerName="istio-proxy"
Oct 30 20:27:18.565788 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.565745    5408 cpu_manager.go:414] "containerMap: removing container" podUID="fbb32dd6-475e-4227-926e-f7eac2ae2ef7" containerName="istio-proxy"
Oct 30 20:27:18.565788 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.565753    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb32dd6-475e-4227-926e-f7eac2ae2ef7" containerName="istio-proxy"
Oct 30 20:27:18.565788 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.565786    5408 cpu_manager.go:414] "containerMap: removing container" podUID="adfef3ed-77c2-4817-9a2c-96403e22f83b" containerName="istio-proxy"
Oct 30 20:27:18.566127 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.565793    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adfef3ed-77c2-4817-9a2c-96403e22f83b" containerName="istio-proxy"
Oct 30 20:27:18.566127 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.565863    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb32dd6-475e-4227-926e-f7eac2ae2ef7" containerName="istio-proxy"
Oct 30 20:27:18.566127 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.565881    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="adfef3ed-77c2-4817-9a2c-96403e22f83b" containerName="istio-proxy"
Oct 30 20:27:18.566127 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.565891    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ba43a0-ddfc-459d-8e82-1c2e197557a0" containerName="istio-proxy"
Oct 30 20:27:18.566127 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.565901    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba388d10-05af-4652-b451-15f429ef1e4d" containerName="istio-proxy"
Oct 30 20:27:18.570313 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.570293    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.600690 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.600660    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"]
Oct 30 20:27:18.649360 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.649330    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fbh7r\""
Oct 30 20:27:18.705254 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.705218    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.705474 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.705281    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.705474 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.705417    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.705474 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.705464    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.705654 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.705485    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.705654 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.705501    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.705654 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.705545    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.705654 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.705646    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ttxf\" (UniqueName: \"kubernetes.io/projected/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-kube-api-access-6ttxf\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.705821 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.705753    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.806531 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.806495    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.806531 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.806539    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.806748 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.806557    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.806748 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.806684    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.806748 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.806715    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.806748 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.806742    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ttxf\" (UniqueName: \"kubernetes.io/projected/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-kube-api-access-6ttxf\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.806970 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.806778    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.806970 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.806848    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.806970 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.806907    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.807184 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.807024    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.807184 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.807124    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.807381 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.807358    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.807506 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.807487    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.807610 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.807590    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.808891 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.808871    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.809345 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.809328    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.824236 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.824181    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.824481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.824463    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ttxf\" (UniqueName: \"kubernetes.io/projected/10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b-kube-api-access-6ttxf\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9nv8z\" (UID: \"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:18.885051 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:18.884995    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:19.170588 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:19.170480    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"]
Oct 30 20:27:19.175691 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:27:19.175664    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10afc8d2_2e5e_4c2f_a3a3_2cb9d2bcf80b.slice/crio-79461bad6fbe226f956520fe8d954ec693f2d6e062b02be57b211e9ed33dbd97 WatchSource:0}: Error finding container 79461bad6fbe226f956520fe8d954ec693f2d6e062b02be57b211e9ed33dbd97: Status 404 returned error can't find the container with id 79461bad6fbe226f956520fe8d954ec693f2d6e062b02be57b211e9ed33dbd97
Oct 30 20:27:19.178222 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:19.178188    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:27:19.178299 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:19.178260    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:27:19.178299 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:19.178287    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:27:19.521923 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:19.521890    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z" event={"ID":"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b","Type":"ContainerStarted","Data":"d0fa56ce9e199fdb2ca16bdf17cc06c4a3ef8bcaf188605ef0484c5aa0308486"}
Oct 30 20:27:19.522116 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:19.521942    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z" event={"ID":"10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b","Type":"ContainerStarted","Data":"79461bad6fbe226f956520fe8d954ec693f2d6e062b02be57b211e9ed33dbd97"}
Oct 30 20:27:19.548922 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:19.548852    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z" podStartSLOduration=1.548834837 podStartE2EDuration="1.548834837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:19.546944785 +0000 UTC m=+3394.131214402" watchObservedRunningTime="2026-10-30 20:27:19.548834837 +0000 UTC m=+3394.133104455"
Oct 30 20:27:19.886117 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:19.886025    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:20.886650 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:20.886616    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-9nv8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Oct 30 20:27:20.887169 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:20.886680    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z" podUID="10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Oct 30 20:27:21.885782 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:21.885749    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-9nv8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Oct 30 20:27:21.885965 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:21.885803    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z" podUID="10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Oct 30 20:27:22.885906 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:22.885865    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-9nv8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Oct 30 20:27:22.886334 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:22.885934    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z" podUID="10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Oct 30 20:27:23.885848 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:23.885813    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-9nv8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Oct 30 20:27:23.886060 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:23.885872    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z" podUID="10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Oct 30 20:27:24.886197 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:24.886160    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-9nv8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Oct 30 20:27:24.886583 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:24.886227    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z" podUID="10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Oct 30 20:27:25.886236 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:25.886202    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-9nv8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Oct 30 20:27:25.886629 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:25.886252    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z" podUID="10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Oct 30 20:27:27.423245 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:27.423206    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:27.424083 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:27.424024    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:27.424833 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:27:27.424811    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9nv8z"
Oct 30 20:27:39.659398 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:27:39.659361    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:28:09.659394 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:28:09.659317    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:28:39.659581 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:28:39.659539    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:28:48.666975 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.666941    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"]
Oct 30 20:28:48.670346 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.670324    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.673673 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.673648    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc905b2-d5e2-4a89-9bbb-435393789d7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.673888 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.673866    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc905b2-d5e2-4a89-9bbb-435393789d7f-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.674298 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.674004    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc905b2-d5e2-4a89-9bbb-435393789d7f-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.674298 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.674067    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm2qf\" (UniqueName: \"kubernetes.io/projected/cfc905b2-d5e2-4a89-9bbb-435393789d7f-kube-api-access-bm2qf\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.674298 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.674103    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc905b2-d5e2-4a89-9bbb-435393789d7f-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.674298 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.674293    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc905b2-d5e2-4a89-9bbb-435393789d7f-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.674614 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.674412    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc905b2-d5e2-4a89-9bbb-435393789d7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.674614 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.674452    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc905b2-d5e2-4a89-9bbb-435393789d7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.674614 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.674477    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc905b2-d5e2-4a89-9bbb-435393789d7f-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.687062 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.687018    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"]
Oct 30 20:28:48.775259 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.775223    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc905b2-d5e2-4a89-9bbb-435393789d7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.775259 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.775260    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc905b2-d5e2-4a89-9bbb-435393789d7f-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.775530 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.775289    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc905b2-d5e2-4a89-9bbb-435393789d7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.775530 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.775415    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc905b2-d5e2-4a89-9bbb-435393789d7f-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.775530 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.775459    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc905b2-d5e2-4a89-9bbb-435393789d7f-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.775530 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.775494    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2qf\" (UniqueName: \"kubernetes.io/projected/cfc905b2-d5e2-4a89-9bbb-435393789d7f-kube-api-access-bm2qf\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.775530 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.775525    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc905b2-d5e2-4a89-9bbb-435393789d7f-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.775827 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.775568    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc905b2-d5e2-4a89-9bbb-435393789d7f-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.775827 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.775637    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc905b2-d5e2-4a89-9bbb-435393789d7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.775827 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.775741    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc905b2-d5e2-4a89-9bbb-435393789d7f-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.775827 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.775820    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc905b2-d5e2-4a89-9bbb-435393789d7f-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.776066 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.776051    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc905b2-d5e2-4a89-9bbb-435393789d7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.776140 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.776055    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc905b2-d5e2-4a89-9bbb-435393789d7f-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.776140 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.776111    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc905b2-d5e2-4a89-9bbb-435393789d7f-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.777822 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.777800    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc905b2-d5e2-4a89-9bbb-435393789d7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.777925 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.777864    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc905b2-d5e2-4a89-9bbb-435393789d7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.788003 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.787982    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc905b2-d5e2-4a89-9bbb-435393789d7f-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.788283 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.788267    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm2qf\" (UniqueName: \"kubernetes.io/projected/cfc905b2-d5e2-4a89-9bbb-435393789d7f-kube-api-access-bm2qf\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zgpln\" (UID: \"cfc905b2-d5e2-4a89-9bbb-435393789d7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:48.984758 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:48.984677    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:49.265260 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:49.265229    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"]
Oct 30 20:28:49.268508 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:28:49.268481    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc905b2_d5e2_4a89_9bbb_435393789d7f.slice/crio-cff3f6319e36d29164d82049b8d3d129e338c218b203a348ca60d580c238ca08 WatchSource:0}: Error finding container cff3f6319e36d29164d82049b8d3d129e338c218b203a348ca60d580c238ca08: Status 404 returned error can't find the container with id cff3f6319e36d29164d82049b8d3d129e338c218b203a348ca60d580c238ca08
Oct 30 20:28:49.270518 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:49.270489    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:28:49.270580 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:49.270553    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:28:49.270618 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:49.270582    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:28:49.898996 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:49.898959    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln" event={"ID":"cfc905b2-d5e2-4a89-9bbb-435393789d7f","Type":"ContainerStarted","Data":"addcf5f7517ec4a80090e35e9ce21522b4940dd332bd54e75938ead9302811fd"}
Oct 30 20:28:49.899533 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:49.899008    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln" event={"ID":"cfc905b2-d5e2-4a89-9bbb-435393789d7f","Type":"ContainerStarted","Data":"cff3f6319e36d29164d82049b8d3d129e338c218b203a348ca60d580c238ca08"}
Oct 30 20:28:49.924385 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:49.924333    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln" podStartSLOduration=1.924318768 podStartE2EDuration="1.924318768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:49.923162619 +0000 UTC m=+3484.507432227" watchObservedRunningTime="2026-10-30 20:28:49.924318768 +0000 UTC m=+3484.508588384"
Oct 30 20:28:49.985611 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:49.985572    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:50.990339 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:50.990306    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:51.907996 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:51.907963    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:28:51.908971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:28:51.908956    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:29:03.680173 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.680140    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"]
Oct 30 20:29:03.683361 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.683344    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.697242 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.697217    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"]
Oct 30 20:29:03.701367 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.701340    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1be84f4-e5e8-4670-b67e-e3064f82f544-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.701497 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.701384    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1be84f4-e5e8-4670-b67e-e3064f82f544-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.701497 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.701463    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1be84f4-e5e8-4670-b67e-e3064f82f544-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.701613 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.701511    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1be84f4-e5e8-4670-b67e-e3064f82f544-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.701668 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.701632    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1be84f4-e5e8-4670-b67e-e3064f82f544-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.701780 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.701673    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1be84f4-e5e8-4670-b67e-e3064f82f544-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.701780 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.701728    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1be84f4-e5e8-4670-b67e-e3064f82f544-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.701780 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.701776    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1be84f4-e5e8-4670-b67e-e3064f82f544-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.701937 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.701809    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjs7n\" (UniqueName: \"kubernetes.io/projected/e1be84f4-e5e8-4670-b67e-e3064f82f544-kube-api-access-sjs7n\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.802412 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.802378    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1be84f4-e5e8-4670-b67e-e3064f82f544-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.802615 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.802457    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1be84f4-e5e8-4670-b67e-e3064f82f544-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.802615 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.802493    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1be84f4-e5e8-4670-b67e-e3064f82f544-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.802615 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.802515    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1be84f4-e5e8-4670-b67e-e3064f82f544-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.802615 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.802538    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1be84f4-e5e8-4670-b67e-e3064f82f544-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.802615 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.802566    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1be84f4-e5e8-4670-b67e-e3064f82f544-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.802910 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.802604    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjs7n\" (UniqueName: \"kubernetes.io/projected/e1be84f4-e5e8-4670-b67e-e3064f82f544-kube-api-access-sjs7n\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.802910 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.802807    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1be84f4-e5e8-4670-b67e-e3064f82f544-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.802910 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.802819    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1be84f4-e5e8-4670-b67e-e3064f82f544-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.802910 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.802865    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1be84f4-e5e8-4670-b67e-e3064f82f544-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.803309 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.803271    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1be84f4-e5e8-4670-b67e-e3064f82f544-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.803401 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.803346    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1be84f4-e5e8-4670-b67e-e3064f82f544-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.803401 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.803363    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1be84f4-e5e8-4670-b67e-e3064f82f544-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.803401 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.803347    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1be84f4-e5e8-4670-b67e-e3064f82f544-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.805066 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.805046    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1be84f4-e5e8-4670-b67e-e3064f82f544-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.805152 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.805091    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1be84f4-e5e8-4670-b67e-e3064f82f544-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.815078 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.815054    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1be84f4-e5e8-4670-b67e-e3064f82f544-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.815180 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.815163    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjs7n\" (UniqueName: \"kubernetes.io/projected/e1be84f4-e5e8-4670-b67e-e3064f82f544-kube-api-access-sjs7n\") pod \"maas-default-gateway-openshift-default-57f78bcd55-j88mg\" (UID: \"e1be84f4-e5e8-4670-b67e-e3064f82f544\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:03.995855 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:03.995828    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:04.281483 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:04.281402    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"]
Oct 30 20:29:04.285287 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:29:04.285248    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1be84f4_e5e8_4670_b67e_e3064f82f544.slice/crio-b4374f807b0b60e3b138fba3d0b99bdb8291e07ec488791328ed661219708eb4 WatchSource:0}: Error finding container b4374f807b0b60e3b138fba3d0b99bdb8291e07ec488791328ed661219708eb4: Status 404 returned error can't find the container with id b4374f807b0b60e3b138fba3d0b99bdb8291e07ec488791328ed661219708eb4
Oct 30 20:29:04.287557 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:04.287524    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:29:04.287633 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:04.287595    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:29:04.287633 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:04.287624    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:29:04.959359 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:04.959327    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg" event={"ID":"e1be84f4-e5e8-4670-b67e-e3064f82f544","Type":"ContainerStarted","Data":"095bbcbc62fe6bb200d400a908bcb51ae8f869b57f946914309724131e3e0da9"}
Oct 30 20:29:04.959779 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:04.959377    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg" event={"ID":"e1be84f4-e5e8-4670-b67e-e3064f82f544","Type":"ContainerStarted","Data":"b4374f807b0b60e3b138fba3d0b99bdb8291e07ec488791328ed661219708eb4"}
Oct 30 20:29:04.987426 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:04.987370    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg" podStartSLOduration=1.987356094 podStartE2EDuration="1.987356094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:04.985108592 +0000 UTC m=+3499.569378211" watchObservedRunningTime="2026-10-30 20:29:04.987356094 +0000 UTC m=+3499.571625711"
Oct 30 20:29:04.996760 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:04.996732    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:06.001978 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:06.001950    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:06.966745 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:06.966718    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:06.967885 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:29:06.967863    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:29:09.658691 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:29:09.658656    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:29:39.660986 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:29:39.660945    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:30:09.660317 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:30:09.660272    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:30:13.276200 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:13.276163    5408 generic.go:356] "Generic (PLEG): container finished" podID="e1be84f4-e5e8-4670-b67e-e3064f82f544" containerID="095bbcbc62fe6bb200d400a908bcb51ae8f869b57f946914309724131e3e0da9" exitCode=0
Oct 30 20:30:13.276634 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:13.276239    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg" event={"ID":"e1be84f4-e5e8-4670-b67e-e3064f82f544","Type":"ContainerDied","Data":"095bbcbc62fe6bb200d400a908bcb51ae8f869b57f946914309724131e3e0da9"}
Oct 30 20:30:13.276682 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:13.276655    5408 scope.go:122] "RemoveContainer" containerID="095bbcbc62fe6bb200d400a908bcb51ae8f869b57f946914309724131e3e0da9"
Oct 30 20:30:13.277845 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:13.277810    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:30:13.277946 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:13.277906    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:30:13.277946 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:13.277937    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:30:14.282815 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:14.282777    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg" event={"ID":"e1be84f4-e5e8-4670-b67e-e3064f82f544","Type":"ContainerStarted","Data":"695745854ff9131b47199999ffb3bcc60b5714060107644cd39364c81291c2b8"}
Oct 30 20:30:14.284586 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:14.284561    5408 generic.go:356] "Generic (PLEG): container finished" podID="cfc905b2-d5e2-4a89-9bbb-435393789d7f" containerID="addcf5f7517ec4a80090e35e9ce21522b4940dd332bd54e75938ead9302811fd" exitCode=0
Oct 30 20:30:14.284708 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:14.284596    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln" event={"ID":"cfc905b2-d5e2-4a89-9bbb-435393789d7f","Type":"ContainerDied","Data":"addcf5f7517ec4a80090e35e9ce21522b4940dd332bd54e75938ead9302811fd"}
Oct 30 20:30:14.284910 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:14.284897    5408 scope.go:122] "RemoveContainer" containerID="addcf5f7517ec4a80090e35e9ce21522b4940dd332bd54e75938ead9302811fd"
Oct 30 20:30:14.285988 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:14.285962    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:30:14.286061 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:14.286048    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:30:14.286114 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:14.286089    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:30:14.996796 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:14.996759    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:30:14.997009 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:14.996988    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-j88mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 20:30:14.997101 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:14.997051    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg" podUID="e1be84f4-e5e8-4670-b67e-e3064f82f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 20:30:15.290480 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:15.290393    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln" event={"ID":"cfc905b2-d5e2-4a89-9bbb-435393789d7f","Type":"ContainerStarted","Data":"50c5314f8f346a1e166e6bff6584c70f489e2009bb514dba8a7fccd773d9713a"}
Oct 30 20:30:15.985103 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:15.985062    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:30:15.985381 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:15.985358    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgpln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 20:30:15.985483 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:15.985411    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln" podUID="cfc905b2-d5e2-4a89-9bbb-435393789d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 20:30:15.996334 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:15.996311    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-j88mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 20:30:15.996431 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:15.996348    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg" podUID="e1be84f4-e5e8-4670-b67e-e3064f82f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 20:30:16.985461 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:16.985426    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgpln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 20:30:16.985965 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:16.985480    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln" podUID="cfc905b2-d5e2-4a89-9bbb-435393789d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 20:30:16.996752 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:16.996725    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-j88mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 20:30:16.996893 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:16.996772    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg" podUID="e1be84f4-e5e8-4670-b67e-e3064f82f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 20:30:17.985129 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:17.985093    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgpln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 20:30:17.985318 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:17.985143    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln" podUID="cfc905b2-d5e2-4a89-9bbb-435393789d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 20:30:17.996296 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:17.996273    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-j88mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 20:30:17.996599 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:17.996318    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg" podUID="e1be84f4-e5e8-4670-b67e-e3064f82f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 20:30:18.985350 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:18.985318    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgpln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 20:30:18.985517 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:18.985372    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln" podUID="cfc905b2-d5e2-4a89-9bbb-435393789d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 20:30:18.996695 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:18.996657    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-j88mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 20:30:18.997065 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:18.996715    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg" podUID="e1be84f4-e5e8-4670-b67e-e3064f82f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 20:30:19.985796 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:19.985757    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgpln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 20:30:19.986019 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:19.985812    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln" podUID="cfc905b2-d5e2-4a89-9bbb-435393789d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 20:30:19.996585 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:19.996553    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-j88mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 20:30:19.996706 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:19.996608    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg" podUID="e1be84f4-e5e8-4670-b67e-e3064f82f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 20:30:20.985695 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:20.985648    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgpln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 20:30:20.986134 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:20.985724    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln" podUID="cfc905b2-d5e2-4a89-9bbb-435393789d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 20:30:21.908340 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:21.908292    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:30:21.966669 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:21.966618    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:30:21.996718 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:21.996677    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-j88mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:30:21.997404 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:21.997360    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg" podUID="e1be84f4-e5e8-4670-b67e-e3064f82f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:30:22.985504 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:22.985461    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgpln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:30:22.985781 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:22.985746    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln" podUID="cfc905b2-d5e2-4a89-9bbb-435393789d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:30:23.038327 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:23.038251    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-j88mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:30:23.039618 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:23.038996    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg" podUID="e1be84f4-e5e8-4670-b67e-e3064f82f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:30:24.031837 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:24.031111    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgpln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:30:24.031837 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:24.031176    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln" podUID="cfc905b2-d5e2-4a89-9bbb-435393789d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:30:24.089291 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:24.089188    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-j88mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:30:24.089291 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:24.089250    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg" podUID="e1be84f4-e5e8-4670-b67e-e3064f82f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:30:25.072266 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:25.072216    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgpln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:30:25.072469 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:25.072304    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln" podUID="cfc905b2-d5e2-4a89-9bbb-435393789d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:30:25.090400 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:25.090356    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-j88mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:30:25.090821 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:25.090417    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg" podUID="e1be84f4-e5e8-4670-b67e-e3064f82f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:30:26.113234 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:26.113191    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgpln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:30:26.114003 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:26.113862    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln" podUID="cfc905b2-d5e2-4a89-9bbb-435393789d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:30:26.133243 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:26.133187    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-j88mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:30:26.133485 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:26.133250    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg" podUID="e1be84f4-e5e8-4670-b67e-e3064f82f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:30:26.564989 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:26.564946    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:30:26.566168 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:26.566147    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg"
Oct 30 20:30:27.156249 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:27.156196    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zgpln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:30:27.156819 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:27.156257    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln" podUID="cfc905b2-d5e2-4a89-9bbb-435393789d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:30:27.399317 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:27.399287    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:30:27.400317 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:27.400295    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zgpln"
Oct 30 20:30:39.659226 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:30:39.659194    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:30:59.997936 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:30:59.997856    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"]
Oct 30 20:31:00.001642 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.001619    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.010145 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.010067    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-de61d2\"/\"openshift-service-ca.crt\""
Oct 30 20:31:00.010145 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.010087    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-de61d2\"/\"kube-root-ca.crt\""
Oct 30 20:31:00.011502 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.011478    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-de61d2\"/\"simulator-single-main-de61d2-kserve-self-signed-certs\""
Oct 30 20:31:00.011502 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.011493    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-de61d2\"/\"default-dockercfg-5qm5q\""
Oct 30 20:31:00.020060 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.020014    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"]
Oct 30 20:31:00.098646 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.098611    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-dshm\") pod \"simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") " pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.098646 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.098651    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-model-cache\") pod \"simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") " pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.098883 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.098682    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-home\") pod \"simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") " pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.098883 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.098743    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c38993f-58bf-491e-9f6f-87f431562422-tls-certs\") pod \"simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") " pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.098883 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.098780    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnv5k\" (UniqueName: \"kubernetes.io/projected/9c38993f-58bf-491e-9f6f-87f431562422-kube-api-access-dnv5k\") pod \"simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") " pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.098883 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.098873    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-tmp-dir\") pod \"simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") " pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.199574 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.199536    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-tmp-dir\") pod \"simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") " pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.199775 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.199613    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-dshm\") pod \"simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") " pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.199775 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.199689    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-model-cache\") pod \"simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") " pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.199775 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.199727    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-home\") pod \"simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") " pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.199935 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.199760    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c38993f-58bf-491e-9f6f-87f431562422-tls-certs\") pod \"simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") " pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.199935 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.199826    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnv5k\" (UniqueName: \"kubernetes.io/projected/9c38993f-58bf-491e-9f6f-87f431562422-kube-api-access-dnv5k\") pod \"simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") " pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.200512 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.200488    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-tmp-dir\") pod \"simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") " pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.200975 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.200949    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-home\") pod \"simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") " pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.201169 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.201060    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-model-cache\") pod \"simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") " pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.203082 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.203057    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-dshm\") pod \"simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") " pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.203551 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.203535    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c38993f-58bf-491e-9f6f-87f431562422-tls-certs\") pod \"simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") " pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.212746 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.212724    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnv5k\" (UniqueName: \"kubernetes.io/projected/9c38993f-58bf-491e-9f6f-87f431562422-kube-api-access-dnv5k\") pod \"simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") " pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.314905 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.314829    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:00.592005 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:00.591926    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"]
Oct 30 20:31:00.595735 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:31:00.595705    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c38993f_58bf_491e_9f6f_87f431562422.slice/crio-1341b5910b824cd6ad520a44ef29a4df61388b99911c9e332f62df88e95256a6 WatchSource:0}: Error finding container 1341b5910b824cd6ad520a44ef29a4df61388b99911c9e332f62df88e95256a6: Status 404 returned error can't find the container with id 1341b5910b824cd6ad520a44ef29a4df61388b99911c9e332f62df88e95256a6
Oct 30 20:31:01.514942 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:01.514892    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2" event={"ID":"9c38993f-58bf-491e-9f6f-87f431562422","Type":"ContainerStarted","Data":"82d5f970add308aa17956560bfa38c94b6ac3de5ef87d6cc3f4ad550fd5c21de"}
Oct 30 20:31:01.514942 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:01.514949    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2" event={"ID":"9c38993f-58bf-491e-9f6f-87f431562422","Type":"ContainerStarted","Data":"1341b5910b824cd6ad520a44ef29a4df61388b99911c9e332f62df88e95256a6"}
Oct 30 20:31:01.537526 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:01.537473    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2" podStartSLOduration=2.537457293 podStartE2EDuration="2.537457293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:01.537162996 +0000 UTC m=+3616.121432617" watchObservedRunningTime="2026-10-30 20:31:01.537457293 +0000 UTC m=+3616.121726911"
Oct 30 20:31:02.315895 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:02.315862    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:09.660983 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:31:09.660943    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:31:10.315146 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:10.315113    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:10.327690 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:10.327663    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:10.564978 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:10.564947    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:31:39.660004 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:31:39.659921    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:31:46.277912 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:31:46.277874    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"]
Oct 30 20:32:01.298460 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.298396    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2" podUID="9c38993f-58bf-491e-9f6f-87f431562422" containerName="main" containerID="cri-o://82d5f970add308aa17956560bfa38c94b6ac3de5ef87d6cc3f4ad550fd5c21de" gracePeriod=45
Oct 30 20:32:01.536316 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.536293    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:32:01.638074 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.637984    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c38993f-58bf-491e-9f6f-87f431562422-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c38993f-58bf-491e-9f6f-87f431562422-tls-certs\") pod \"9c38993f-58bf-491e-9f6f-87f431562422\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") "
Oct 30 20:32:01.638074 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.638071    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-dshm\") pod \"9c38993f-58bf-491e-9f6f-87f431562422\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") "
Oct 30 20:32:01.638295 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.638105    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-tmp-dir\") pod \"9c38993f-58bf-491e-9f6f-87f431562422\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") "
Oct 30 20:32:01.638295 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.638154    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c38993f-58bf-491e-9f6f-87f431562422-kube-api-access-dnv5k\" (UniqueName: \"kubernetes.io/projected/9c38993f-58bf-491e-9f6f-87f431562422-kube-api-access-dnv5k\") pod \"9c38993f-58bf-491e-9f6f-87f431562422\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") "
Oct 30 20:32:01.638295 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.638267    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-home\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-home\") pod \"9c38993f-58bf-491e-9f6f-87f431562422\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") "
Oct 30 20:32:01.638482 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.638325    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-model-cache\") pod \"9c38993f-58bf-491e-9f6f-87f431562422\" (UID: \"9c38993f-58bf-491e-9f6f-87f431562422\") "
Oct 30 20:32:01.638482 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.638454    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-tmp-dir" pod "9c38993f-58bf-491e-9f6f-87f431562422" (UID: "9c38993f-58bf-491e-9f6f-87f431562422"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:32:01.638900 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.638659    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-model-cache" pod "9c38993f-58bf-491e-9f6f-87f431562422" (UID: "9c38993f-58bf-491e-9f6f-87f431562422"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:32:01.638900 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.638710    5408 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-tmp-dir\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:01.638900 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.638838    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-home" pod "9c38993f-58bf-491e-9f6f-87f431562422" (UID: "9c38993f-58bf-491e-9f6f-87f431562422"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:32:01.640424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.640400    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c38993f-58bf-491e-9f6f-87f431562422-tls-certs" pod "9c38993f-58bf-491e-9f6f-87f431562422" (UID: "9c38993f-58bf-491e-9f6f-87f431562422"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:32:01.640580 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.640558    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-dshm" pod "9c38993f-58bf-491e-9f6f-87f431562422" (UID: "9c38993f-58bf-491e-9f6f-87f431562422"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:32:01.640798 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.640776    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c38993f-58bf-491e-9f6f-87f431562422-kube-api-access-dnv5k" pod "9c38993f-58bf-491e-9f6f-87f431562422" (UID: "9c38993f-58bf-491e-9f6f-87f431562422"). InnerVolumeSpecName "kube-api-access-dnv5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:32:01.739469 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.739442    5408 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-home\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:01.739469 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.739469    5408 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-model-cache\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:01.739646 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.739481    5408 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c38993f-58bf-491e-9f6f-87f431562422-tls-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:01.739646 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.739490    5408 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c38993f-58bf-491e-9f6f-87f431562422-dshm\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:01.739646 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.739498    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnv5k\" (UniqueName: \"kubernetes.io/projected/9c38993f-58bf-491e-9f6f-87f431562422-kube-api-access-dnv5k\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:01.785746 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.785719    5408 generic.go:356] "Generic (PLEG): container finished" podID="9c38993f-58bf-491e-9f6f-87f431562422" containerID="82d5f970add308aa17956560bfa38c94b6ac3de5ef87d6cc3f4ad550fd5c21de" exitCode=0
Oct 30 20:32:01.785863 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.785762    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"
Oct 30 20:32:01.785863 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.785771    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2" event={"ID":"9c38993f-58bf-491e-9f6f-87f431562422","Type":"ContainerDied","Data":"82d5f970add308aa17956560bfa38c94b6ac3de5ef87d6cc3f4ad550fd5c21de"}
Oct 30 20:32:01.785863 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.785803    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2" event={"ID":"9c38993f-58bf-491e-9f6f-87f431562422","Type":"ContainerDied","Data":"1341b5910b824cd6ad520a44ef29a4df61388b99911c9e332f62df88e95256a6"}
Oct 30 20:32:01.785863 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.785818    5408 scope.go:122] "RemoveContainer" containerID="82d5f970add308aa17956560bfa38c94b6ac3de5ef87d6cc3f4ad550fd5c21de"
Oct 30 20:32:01.794344 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.794323    5408 scope.go:122] "RemoveContainer" containerID="82d5f970add308aa17956560bfa38c94b6ac3de5ef87d6cc3f4ad550fd5c21de"
Oct 30 20:32:01.794584 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:32:01.794568    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d5f970add308aa17956560bfa38c94b6ac3de5ef87d6cc3f4ad550fd5c21de\": container with ID starting with 82d5f970add308aa17956560bfa38c94b6ac3de5ef87d6cc3f4ad550fd5c21de not found: ID does not exist" containerID="82d5f970add308aa17956560bfa38c94b6ac3de5ef87d6cc3f4ad550fd5c21de"
Oct 30 20:32:01.794638 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.794592    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d5f970add308aa17956560bfa38c94b6ac3de5ef87d6cc3f4ad550fd5c21de"} err="failed to get container status \"82d5f970add308aa17956560bfa38c94b6ac3de5ef87d6cc3f4ad550fd5c21de\": rpc error: code = NotFound desc = could not find container \"82d5f970add308aa17956560bfa38c94b6ac3de5ef87d6cc3f4ad550fd5c21de\": container with ID starting with 82d5f970add308aa17956560bfa38c94b6ac3de5ef87d6cc3f4ad550fd5c21de not found: ID does not exist"
Oct 30 20:32:01.807594 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.807572    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"]
Oct 30 20:32:01.813660 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:01.813636    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-de61d2/simulator-single-main-de61d2-kserve-7c6fd5c954-lvct2"]
Oct 30 20:32:03.529863 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:32:03.529832    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c38993f-58bf-491e-9f6f-87f431562422" path="/var/lib/kubelet/pods/9c38993f-58bf-491e-9f6f-87f431562422/volumes"
Oct 30 20:32:09.698858 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:32:09.698819    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:32:39.670594 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:32:39.670500    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:33:09.694191 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:33:09.694156    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:33:39.659529 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:33:39.659492    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:33:43.003017 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:43.002982    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"]
Oct 30 20:33:43.003524 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:43.003281    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t" podUID="057b7980-6217-4c2d-98b4-63e8deda405a" containerName="maas-api" containerID="cri-o://368631c85c69b2235bdfd826f0c88513ea14c7143aec37ac53c53b2db03f56ca" gracePeriod=30
Oct 30 20:33:43.307297 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:43.307203    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t" podUID="057b7980-6217-4c2d-98b4-63e8deda405a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.32:8443/health\": dial tcp 10.134.0.32:8443: connect: connection refused"
Oct 30 20:33:44.252223 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.252201    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"
Oct 30 20:33:44.295351 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.295324    5408 generic.go:356] "Generic (PLEG): container finished" podID="057b7980-6217-4c2d-98b4-63e8deda405a" containerID="368631c85c69b2235bdfd826f0c88513ea14c7143aec37ac53c53b2db03f56ca" exitCode=0
Oct 30 20:33:44.295511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.295366    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"
Oct 30 20:33:44.295511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.295411    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t" event={"ID":"057b7980-6217-4c2d-98b4-63e8deda405a","Type":"ContainerDied","Data":"368631c85c69b2235bdfd826f0c88513ea14c7143aec37ac53c53b2db03f56ca"}
Oct 30 20:33:44.295511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.295472    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-859f666b94-5kb4t" event={"ID":"057b7980-6217-4c2d-98b4-63e8deda405a","Type":"ContainerDied","Data":"605bd3422b82c07f6a1fba488345a2cd5a8866153946810b194f13ccf0fff8fa"}
Oct 30 20:33:44.295511 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.295505    5408 scope.go:122] "RemoveContainer" containerID="368631c85c69b2235bdfd826f0c88513ea14c7143aec37ac53c53b2db03f56ca"
Oct 30 20:33:44.304881 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.304862    5408 scope.go:122] "RemoveContainer" containerID="368631c85c69b2235bdfd826f0c88513ea14c7143aec37ac53c53b2db03f56ca"
Oct 30 20:33:44.305218 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:33:44.305191    5408 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368631c85c69b2235bdfd826f0c88513ea14c7143aec37ac53c53b2db03f56ca\": container with ID starting with 368631c85c69b2235bdfd826f0c88513ea14c7143aec37ac53c53b2db03f56ca not found: ID does not exist" containerID="368631c85c69b2235bdfd826f0c88513ea14c7143aec37ac53c53b2db03f56ca"
Oct 30 20:33:44.305328 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.305231    5408 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368631c85c69b2235bdfd826f0c88513ea14c7143aec37ac53c53b2db03f56ca"} err="failed to get container status \"368631c85c69b2235bdfd826f0c88513ea14c7143aec37ac53c53b2db03f56ca\": rpc error: code = NotFound desc = could not find container \"368631c85c69b2235bdfd826f0c88513ea14c7143aec37ac53c53b2db03f56ca\": container with ID starting with 368631c85c69b2235bdfd826f0c88513ea14c7143aec37ac53c53b2db03f56ca not found: ID does not exist"
Oct 30 20:33:44.337947 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.337892    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/057b7980-6217-4c2d-98b4-63e8deda405a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/057b7980-6217-4c2d-98b4-63e8deda405a-maas-api-tls\") pod \"057b7980-6217-4c2d-98b4-63e8deda405a\" (UID: \"057b7980-6217-4c2d-98b4-63e8deda405a\") "
Oct 30 20:33:44.337947 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.337934    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/057b7980-6217-4c2d-98b4-63e8deda405a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/057b7980-6217-4c2d-98b4-63e8deda405a-openshift-service-ca\") pod \"057b7980-6217-4c2d-98b4-63e8deda405a\" (UID: \"057b7980-6217-4c2d-98b4-63e8deda405a\") "
Oct 30 20:33:44.338138 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.337959    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/057b7980-6217-4c2d-98b4-63e8deda405a-kube-api-access-kzjfr\" (UniqueName: \"kubernetes.io/projected/057b7980-6217-4c2d-98b4-63e8deda405a-kube-api-access-kzjfr\") pod \"057b7980-6217-4c2d-98b4-63e8deda405a\" (UID: \"057b7980-6217-4c2d-98b4-63e8deda405a\") "
Oct 30 20:33:44.338138 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.338100    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/057b7980-6217-4c2d-98b4-63e8deda405a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/057b7980-6217-4c2d-98b4-63e8deda405a-metrics-certs\") pod \"057b7980-6217-4c2d-98b4-63e8deda405a\" (UID: \"057b7980-6217-4c2d-98b4-63e8deda405a\") "
Oct 30 20:33:44.338597 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.338565    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057b7980-6217-4c2d-98b4-63e8deda405a-openshift-service-ca" pod "057b7980-6217-4c2d-98b4-63e8deda405a" (UID: "057b7980-6217-4c2d-98b4-63e8deda405a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:33:44.340199 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.340174    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057b7980-6217-4c2d-98b4-63e8deda405a-kube-api-access-kzjfr" pod "057b7980-6217-4c2d-98b4-63e8deda405a" (UID: "057b7980-6217-4c2d-98b4-63e8deda405a"). InnerVolumeSpecName "kube-api-access-kzjfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:33:44.340281 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.340183    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/057b7980-6217-4c2d-98b4-63e8deda405a-metrics-certs" pod "057b7980-6217-4c2d-98b4-63e8deda405a" (UID: "057b7980-6217-4c2d-98b4-63e8deda405a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:44.340281 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.340208    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/057b7980-6217-4c2d-98b4-63e8deda405a-maas-api-tls" pod "057b7980-6217-4c2d-98b4-63e8deda405a" (UID: "057b7980-6217-4c2d-98b4-63e8deda405a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:44.439048 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.438995    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/057b7980-6217-4c2d-98b4-63e8deda405a-maas-api-tls\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:44.439048 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.439024    5408 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/057b7980-6217-4c2d-98b4-63e8deda405a-openshift-service-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:44.439048 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.439047    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzjfr\" (UniqueName: \"kubernetes.io/projected/057b7980-6217-4c2d-98b4-63e8deda405a-kube-api-access-kzjfr\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:44.439048 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.439057    5408 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/057b7980-6217-4c2d-98b4-63e8deda405a-metrics-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:44.652269 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.652238    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"]
Oct 30 20:33:44.663574 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:44.663545    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-859f666b94-5kb4t"]
Oct 30 20:33:45.529612 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:45.529582    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="057b7980-6217-4c2d-98b4-63e8deda405a" path="/var/lib/kubelet/pods/057b7980-6217-4c2d-98b4-63e8deda405a/volumes"
Oct 30 20:33:54.557513 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.557472    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"]
Oct 30 20:33:54.557914 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.557830    5408 cpu_manager.go:414] "containerMap: removing container" podUID="9c38993f-58bf-491e-9f6f-87f431562422" containerName="main"
Oct 30 20:33:54.557914 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.557842    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c38993f-58bf-491e-9f6f-87f431562422" containerName="main"
Oct 30 20:33:54.557914 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.557854    5408 cpu_manager.go:414] "containerMap: removing container" podUID="057b7980-6217-4c2d-98b4-63e8deda405a" containerName="maas-api"
Oct 30 20:33:54.557914 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.557859    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057b7980-6217-4c2d-98b4-63e8deda405a" containerName="maas-api"
Oct 30 20:33:54.558222 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.557917    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c38993f-58bf-491e-9f6f-87f431562422" containerName="main"
Oct 30 20:33:54.558222 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.557926    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="057b7980-6217-4c2d-98b4-63e8deda405a" containerName="maas-api"
Oct 30 20:33:54.560798 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.560782    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"
Oct 30 20:33:54.572410 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.572383    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:33:54.573847 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.573808    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:33:54.575385 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.575362    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"]
Oct 30 20:33:54.643529 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.643500    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx6j7\" (UniqueName: \"kubernetes.io/projected/f59bac78-4b94-4875-8976-38e0f538c9d2-kube-api-access-zx6j7\") pod \"maas-api-revoke-keys-e2e-ait-33631e52-87tqk\" (UID: \"f59bac78-4b94-4875-8976-38e0f538c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"
Oct 30 20:33:54.643707 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.643584    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59bac78-4b94-4875-8976-38e0f538c9d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-33631e52-87tqk\" (UID: \"f59bac78-4b94-4875-8976-38e0f538c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"
Oct 30 20:33:54.643707 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.643622    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59bac78-4b94-4875-8976-38e0f538c9d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-33631e52-87tqk\" (UID: \"f59bac78-4b94-4875-8976-38e0f538c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"
Oct 30 20:33:54.744422 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.744388    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59bac78-4b94-4875-8976-38e0f538c9d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-33631e52-87tqk\" (UID: \"f59bac78-4b94-4875-8976-38e0f538c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"
Oct 30 20:33:54.744620 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.744438    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59bac78-4b94-4875-8976-38e0f538c9d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-33631e52-87tqk\" (UID: \"f59bac78-4b94-4875-8976-38e0f538c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"
Oct 30 20:33:54.744620 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.744530    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx6j7\" (UniqueName: \"kubernetes.io/projected/f59bac78-4b94-4875-8976-38e0f538c9d2-kube-api-access-zx6j7\") pod \"maas-api-revoke-keys-e2e-ait-33631e52-87tqk\" (UID: \"f59bac78-4b94-4875-8976-38e0f538c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"
Oct 30 20:33:54.745161 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.745139    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59bac78-4b94-4875-8976-38e0f538c9d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-33631e52-87tqk\" (UID: \"f59bac78-4b94-4875-8976-38e0f538c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"
Oct 30 20:33:54.746820 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:54.746795    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59bac78-4b94-4875-8976-38e0f538c9d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-33631e52-87tqk\" (UID: \"f59bac78-4b94-4875-8976-38e0f538c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"
Oct 30 20:33:54.751525 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:33:54.751507    5408 projected.go:196] Error preparing data for projected volume kube-api-access-zx6j7 for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 20:33:54.751603 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:33:54.751568    5408 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f59bac78-4b94-4875-8976-38e0f538c9d2-kube-api-access-zx6j7 podName:f59bac78-4b94-4875-8976-38e0f538c9d2 nodeName:}" failed. No retries permitted until 2026-10-30 20:33:55.25155217 +0000 UTC m=+3789.835821769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zx6j7" (UniqueName: "kubernetes.io/projected/f59bac78-4b94-4875-8976-38e0f538c9d2-kube-api-access-zx6j7") pod "maas-api-revoke-keys-e2e-ait-33631e52-87tqk" (UID: "f59bac78-4b94-4875-8976-38e0f538c9d2") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 20:33:55.348957 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:55.348921    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx6j7\" (UniqueName: \"kubernetes.io/projected/f59bac78-4b94-4875-8976-38e0f538c9d2-kube-api-access-zx6j7\") pod \"maas-api-revoke-keys-e2e-ait-33631e52-87tqk\" (UID: \"f59bac78-4b94-4875-8976-38e0f538c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"
Oct 30 20:33:55.362159 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:55.362126    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx6j7\" (UniqueName: \"kubernetes.io/projected/f59bac78-4b94-4875-8976-38e0f538c9d2-kube-api-access-zx6j7\") pod \"maas-api-revoke-keys-e2e-ait-33631e52-87tqk\" (UID: \"f59bac78-4b94-4875-8976-38e0f538c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"
Oct 30 20:33:55.472947 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:55.472912    5408 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk" secret="" err="secret \"maas-api-cleanup-dockercfg-6zmll\" not found"
Oct 30 20:33:55.473154 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:55.472966    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"
Oct 30 20:33:55.747274 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:55.747236    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"]
Oct 30 20:33:55.750995 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:33:55.750966    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59bac78_4b94_4875_8976_38e0f538c9d2.slice/crio-b21d4e4fa0c027e59cc5581a1ebc89dfcf7fd402744c08967c20c2b6e369b2a7 WatchSource:0}: Error finding container b21d4e4fa0c027e59cc5581a1ebc89dfcf7fd402744c08967c20c2b6e369b2a7: Status 404 returned error can't find the container with id b21d4e4fa0c027e59cc5581a1ebc89dfcf7fd402744c08967c20c2b6e369b2a7
Oct 30 20:33:56.350935 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:56.350899    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk" event={"ID":"f59bac78-4b94-4875-8976-38e0f538c9d2","Type":"ContainerStarted","Data":"f67f26c8633f0e390112a13d7f039b57c86ab8b3dbfabdc97d132a30063bd16b"}
Oct 30 20:33:56.351155 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:56.350949    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk" event={"ID":"f59bac78-4b94-4875-8976-38e0f538c9d2","Type":"ContainerStarted","Data":"b21d4e4fa0c027e59cc5581a1ebc89dfcf7fd402744c08967c20c2b6e369b2a7"}
Oct 30 20:33:56.351216 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:56.351201    5408 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk" secret="" err="secret \"maas-api-cleanup-dockercfg-6zmll\" not found"
Oct 30 20:33:56.374304 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:56.374251    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk" podStartSLOduration=2.374235797 podStartE2EDuration="2.374235797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:56.373539123 +0000 UTC m=+3790.957808742" watchObservedRunningTime="2026-10-30 20:33:56.374235797 +0000 UTC m=+3790.958505414"
Oct 30 20:33:57.356202 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:57.356113    5408 generic.go:356] "Generic (PLEG): container finished" podID="f59bac78-4b94-4875-8976-38e0f538c9d2" containerID="f67f26c8633f0e390112a13d7f039b57c86ab8b3dbfabdc97d132a30063bd16b" exitCode=0
Oct 30 20:33:57.356202 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:57.356162    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk" event={"ID":"f59bac78-4b94-4875-8976-38e0f538c9d2","Type":"ContainerDied","Data":"f67f26c8633f0e390112a13d7f039b57c86ab8b3dbfabdc97d132a30063bd16b"}
Oct 30 20:33:58.491140 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:58.491117    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"
Oct 30 20:33:58.572467 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:58.572439    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f59bac78-4b94-4875-8976-38e0f538c9d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59bac78-4b94-4875-8976-38e0f538c9d2-maas-api-openshift-ca\") pod \"f59bac78-4b94-4875-8976-38e0f538c9d2\" (UID: \"f59bac78-4b94-4875-8976-38e0f538c9d2\") "
Oct 30 20:33:58.572638 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:58.572537    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f59bac78-4b94-4875-8976-38e0f538c9d2-kube-api-access-zx6j7\" (UniqueName: \"kubernetes.io/projected/f59bac78-4b94-4875-8976-38e0f538c9d2-kube-api-access-zx6j7\") pod \"f59bac78-4b94-4875-8976-38e0f538c9d2\" (UID: \"f59bac78-4b94-4875-8976-38e0f538c9d2\") "
Oct 30 20:33:58.572638 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:58.572576    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f59bac78-4b94-4875-8976-38e0f538c9d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59bac78-4b94-4875-8976-38e0f538c9d2-maas-api-xks-ca\") pod \"f59bac78-4b94-4875-8976-38e0f538c9d2\" (UID: \"f59bac78-4b94-4875-8976-38e0f538c9d2\") "
Oct 30 20:33:58.572942 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:58.572838    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59bac78-4b94-4875-8976-38e0f538c9d2-maas-api-openshift-ca" pod "f59bac78-4b94-4875-8976-38e0f538c9d2" (UID: "f59bac78-4b94-4875-8976-38e0f538c9d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:33:58.575274 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:58.575240    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59bac78-4b94-4875-8976-38e0f538c9d2-kube-api-access-zx6j7" pod "f59bac78-4b94-4875-8976-38e0f538c9d2" (UID: "f59bac78-4b94-4875-8976-38e0f538c9d2"). InnerVolumeSpecName "kube-api-access-zx6j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:33:58.575501 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:58.575466    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f59bac78-4b94-4875-8976-38e0f538c9d2-maas-api-xks-ca" pod "f59bac78-4b94-4875-8976-38e0f538c9d2" (UID: "f59bac78-4b94-4875-8976-38e0f538c9d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:58.674007 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:58.673942    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx6j7\" (UniqueName: \"kubernetes.io/projected/f59bac78-4b94-4875-8976-38e0f538c9d2-kube-api-access-zx6j7\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:58.674007 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:58.673967    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59bac78-4b94-4875-8976-38e0f538c9d2-maas-api-xks-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:58.674007 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:58.673977    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59bac78-4b94-4875-8976-38e0f538c9d2-maas-api-openshift-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:59.366590 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:59.366563    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"
Oct 30 20:33:59.366801 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:59.366558    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk" event={"ID":"f59bac78-4b94-4875-8976-38e0f538c9d2","Type":"ContainerDied","Data":"b21d4e4fa0c027e59cc5581a1ebc89dfcf7fd402744c08967c20c2b6e369b2a7"}
Oct 30 20:33:59.366937 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:33:59.366851    5408 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b21d4e4fa0c027e59cc5581a1ebc89dfcf7fd402744c08967c20c2b6e369b2a7"
Oct 30 20:34:06.104893 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:06.104863    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"]
Oct 30 20:34:06.106981 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:06.106955    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33631e52-87tqk"]
Oct 30 20:34:07.529681 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:07.529646    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f59bac78-4b94-4875-8976-38e0f538c9d2" path="/var/lib/kubelet/pods/f59bac78-4b94-4875-8976-38e0f538c9d2/volumes"
Oct 30 20:34:09.659451 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:34:09.659411    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:34:12.835745 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:12.835710    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"]
Oct 30 20:34:12.836331 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:12.836104    5408 cpu_manager.go:414] "containerMap: removing container" podUID="f59bac78-4b94-4875-8976-38e0f538c9d2" containerName="revoke-keys"
Oct 30 20:34:12.836331 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:12.836119    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f59bac78-4b94-4875-8976-38e0f538c9d2" containerName="revoke-keys"
Oct 30 20:34:12.836331 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:12.836216    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="f59bac78-4b94-4875-8976-38e0f538c9d2" containerName="revoke-keys"
Oct 30 20:34:12.839598 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:12.839576    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:12.846514 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:12.846488    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-48059e-openshift-default-dockercfg-flgmf\""
Oct 30 20:34:12.853994 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:12.853956    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"]
Oct 30 20:34:12.909908 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:12.909875    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-envoy\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:12.909908 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:12.909911    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-token\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:12.910155 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:12.909930    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-podinfo\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:12.910155 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:12.910003    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-workload-socket\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:12.910155 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:12.910144    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-workload-certs\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:12.910289 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:12.910192    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9219bcf1-acbc-4725-95bd-791768bcbd0b-istiod-ca-cert\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:12.910339 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:12.910284    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk62q\" (UniqueName: \"kubernetes.io/projected/9219bcf1-acbc-4725-95bd-791768bcbd0b-kube-api-access-kk62q\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:12.910339 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:12.910321    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-data\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:12.910431 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:12.910416    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-credential-socket\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.011285 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.011243    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk62q\" (UniqueName: \"kubernetes.io/projected/9219bcf1-acbc-4725-95bd-791768bcbd0b-kube-api-access-kk62q\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.011504 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.011305    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-data\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.011504 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.011386    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-credential-socket\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.011504 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.011414    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-envoy\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.011504 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.011441    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-token\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.011504 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.011471    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-podinfo\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.011504 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.011500    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-workload-socket\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.011867 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.011542    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-workload-certs\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.011867 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.011696    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9219bcf1-acbc-4725-95bd-791768bcbd0b-istiod-ca-cert\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.011867 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.011846    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-credential-socket\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.012147 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.011877    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-data\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.012610 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.012583    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-workload-certs\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.012754 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.012734    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9219bcf1-acbc-4725-95bd-791768bcbd0b-istiod-ca-cert\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.012833 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.012738    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-workload-socket\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.025592 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.025564    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk62q\" (UniqueName: \"kubernetes.io/projected/9219bcf1-acbc-4725-95bd-791768bcbd0b-kube-api-access-kk62q\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.025869 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.025845    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-envoy\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.025926 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.025889    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-podinfo\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.027571 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.027546    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-token\") pod \"e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") " pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.153833 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.153740    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:13.435541 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.435467    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"]
Oct 30 20:34:13.438517 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:34:13.438486    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9219bcf1_acbc_4725_95bd_791768bcbd0b.slice/crio-bb0d56742f3c826e339a13861bd32b04d94380127bbe0ee67e063e866b9deae9 WatchSource:0}: Error finding container bb0d56742f3c826e339a13861bd32b04d94380127bbe0ee67e063e866b9deae9: Status 404 returned error can't find the container with id bb0d56742f3c826e339a13861bd32b04d94380127bbe0ee67e063e866b9deae9
Oct 30 20:34:13.440629 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.440594    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:34:13.440713 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.440660    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:34:13.440713 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:13.440688    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:34:14.431992 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:14.431952    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt" event={"ID":"9219bcf1-acbc-4725-95bd-791768bcbd0b","Type":"ContainerStarted","Data":"c58908195b631eb03e47376acc87347b9a8b4410b061b753ed32599cd897b7fe"}
Oct 30 20:34:14.432425 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:14.432009    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt" event={"ID":"9219bcf1-acbc-4725-95bd-791768bcbd0b","Type":"ContainerStarted","Data":"bb0d56742f3c826e339a13861bd32b04d94380127bbe0ee67e063e866b9deae9"}
Oct 30 20:34:14.462681 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:14.462626    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt" podStartSLOduration=2.462612899 podStartE2EDuration="2.462612899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:14.460388052 +0000 UTC m=+3809.044657670" watchObservedRunningTime="2026-10-30 20:34:14.462612899 +0000 UTC m=+3809.046882611"
Oct 30 20:34:15.154710 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:15.154670    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:15.159659 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:15.159633    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:15.436286 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:15.436192    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:15.437111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:15.437088    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:30.438109 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:30.438071    5408 patch_prober.go:29] interesting pod/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 30 20:34:30.438563 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:30.438137    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt" podUID="9219bcf1-acbc-4725-95bd-791768bcbd0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 30 20:34:30.509226 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:30.509193    5408 generic.go:356] "Generic (PLEG): container finished" podID="9219bcf1-acbc-4725-95bd-791768bcbd0b" containerID="c58908195b631eb03e47376acc87347b9a8b4410b061b753ed32599cd897b7fe" exitCode=0
Oct 30 20:34:30.509422 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:30.509250    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt" event={"ID":"9219bcf1-acbc-4725-95bd-791768bcbd0b","Type":"ContainerDied","Data":"c58908195b631eb03e47376acc87347b9a8b4410b061b753ed32599cd897b7fe"}
Oct 30 20:34:30.509612 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:30.509595    5408 scope.go:122] "RemoveContainer" containerID="c58908195b631eb03e47376acc87347b9a8b4410b061b753ed32599cd897b7fe"
Oct 30 20:34:30.510692 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:30.510661    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:34:30.510783 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:30.510729    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:34:30.510783 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:30.510759    5408 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:34:31.537714 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:31.537678    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt" event={"ID":"9219bcf1-acbc-4725-95bd-791768bcbd0b","Type":"ContainerStarted","Data":"be132da32f12936eeabcb7426e6c90d6b3f216daca368b97cf98251a368b3cee"}
Oct 30 20:34:32.154012 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:32.153982    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:32.154319 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:32.154288    5408 patch_prober.go:29] interesting pod/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 30 20:34:32.154443 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:32.154346    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt" podUID="9219bcf1-acbc-4725-95bd-791768bcbd0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 30 20:34:33.154401 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:33.154365    5408 patch_prober.go:29] interesting pod/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 30 20:34:33.154810 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:33.154418    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt" podUID="9219bcf1-acbc-4725-95bd-791768bcbd0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 30 20:34:34.154986 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:34.154934    5408 patch_prober.go:29] interesting pod/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 30 20:34:34.155469 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:34.155004    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt" podUID="9219bcf1-acbc-4725-95bd-791768bcbd0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 30 20:34:35.154206 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:35.154173    5408 patch_prober.go:29] interesting pod/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 30 20:34:35.154406 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:35.154230    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt" podUID="9219bcf1-acbc-4725-95bd-791768bcbd0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 30 20:34:36.154906 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:36.154872    5408 patch_prober.go:29] interesting pod/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 30 20:34:36.155325 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:36.154937    5408 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt" podUID="9219bcf1-acbc-4725-95bd-791768bcbd0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 30 20:34:37.158413 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:37.158378    5408 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:37.158808 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:37.158641    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:37.159385 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:34:37.159366    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:34:39.660139 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:34:39.660103    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:35:09.659169 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:35:09.659131    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:35:39.660245 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:35:39.660154    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:35:52.376767 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:52.376733    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf"]
Oct 30 20:35:52.380055 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:52.380014    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf"
Oct 30 20:35:52.387065 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:52.386708    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:35:52.387065 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:52.386713    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-khrtr\""
Oct 30 20:35:52.390069 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:52.390019    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf"]
Oct 30 20:35:52.477510 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:52.477476    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw4mn\" (UniqueName: \"kubernetes.io/projected/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-kube-api-access-cw4mn\") pod \"maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf\" (UID: \"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf"
Oct 30 20:35:52.477686 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:52.477611    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf\" (UID: \"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf"
Oct 30 20:35:52.477686 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:52.477654    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf\" (UID: \"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf"
Oct 30 20:35:52.578349 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:52.578319    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf\" (UID: \"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf"
Oct 30 20:35:52.578538 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:52.578379    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf\" (UID: \"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf"
Oct 30 20:35:52.578538 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:52.578405    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw4mn\" (UniqueName: \"kubernetes.io/projected/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-kube-api-access-cw4mn\") pod \"maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf\" (UID: \"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf"
Oct 30 20:35:52.579131 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:52.579110    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf\" (UID: \"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf"
Oct 30 20:35:52.580709 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:52.580687    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf\" (UID: \"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf"
Oct 30 20:35:52.591639 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:52.591609    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw4mn\" (UniqueName: \"kubernetes.io/projected/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-kube-api-access-cw4mn\") pod \"maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf\" (UID: \"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf"
Oct 30 20:35:52.705475 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:52.705399    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf"
Oct 30 20:35:52.983803 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:52.983777    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf"]
Oct 30 20:35:52.986731 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:35:52.986704    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a5bc2d_74b6_4fb4_a0fa_3787dee0450f.slice/crio-2788a90f22cab554854cf5dfd5353cd79fc2feba600c4d8075332e8bbb2b40d6 WatchSource:0}: Error finding container 2788a90f22cab554854cf5dfd5353cd79fc2feba600c4d8075332e8bbb2b40d6: Status 404 returned error can't find the container with id 2788a90f22cab554854cf5dfd5353cd79fc2feba600c4d8075332e8bbb2b40d6
Oct 30 20:35:53.908092 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:53.908063    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf" event={"ID":"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f","Type":"ContainerStarted","Data":"2ba75c02dc7b569fce34b25554681d1c2b0b759cb4449d2fa0a7618499a1615c"}
Oct 30 20:35:53.908529 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:53.908125    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf" event={"ID":"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f","Type":"ContainerStarted","Data":"2788a90f22cab554854cf5dfd5353cd79fc2feba600c4d8075332e8bbb2b40d6"}
Oct 30 20:35:53.929990 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:53.929947    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf" podStartSLOduration=1.9299318909999998 podStartE2EDuration="1.929931891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:53.927824466 +0000 UTC m=+3908.512094083" watchObservedRunningTime="2026-10-30 20:35:53.929931891 +0000 UTC m=+3908.514201554"
Oct 30 20:35:54.915143 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:54.915104    5408 generic.go:356] "Generic (PLEG): container finished" podID="d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f" containerID="2ba75c02dc7b569fce34b25554681d1c2b0b759cb4449d2fa0a7618499a1615c" exitCode=0
Oct 30 20:35:54.915678 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:54.915173    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf" event={"ID":"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f","Type":"ContainerDied","Data":"2ba75c02dc7b569fce34b25554681d1c2b0b759cb4449d2fa0a7618499a1615c"}
Oct 30 20:35:56.061446 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:56.061420    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf"
Oct 30 20:35:56.217329 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:56.217240    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-maas-api-openshift-ca\") pod \"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f\" (UID: \"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f\") "
Oct 30 20:35:56.217329 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:56.217311    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-kube-api-access-cw4mn\" (UniqueName: \"kubernetes.io/projected/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-kube-api-access-cw4mn\") pod \"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f\" (UID: \"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f\") "
Oct 30 20:35:56.217525 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:56.217414    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-maas-api-xks-ca\") pod \"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f\" (UID: \"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f\") "
Oct 30 20:35:56.217654 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:56.217631    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-maas-api-openshift-ca" pod "d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f" (UID: "d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:56.219433 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:56.219401    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-maas-api-xks-ca" pod "d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f" (UID: "d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:56.219559 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:56.219462    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-kube-api-access-cw4mn" pod "d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f" (UID: "d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f"). InnerVolumeSpecName "kube-api-access-cw4mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:56.318828 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:56.318797    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-maas-api-xks-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:56.318828 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:56.318829    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-maas-api-openshift-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:56.318828 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:56.318842    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw4mn\" (UniqueName: \"kubernetes.io/projected/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f-kube-api-access-cw4mn\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:56.925954 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:56.925920    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf"
Oct 30 20:35:56.926157 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:56.925951    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf" event={"ID":"d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f","Type":"ContainerDied","Data":"2788a90f22cab554854cf5dfd5353cd79fc2feba600c4d8075332e8bbb2b40d6"}
Oct 30 20:35:56.926157 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:35:56.925998    5408 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2788a90f22cab554854cf5dfd5353cd79fc2feba600c4d8075332e8bbb2b40d6"
Oct 30 20:36:08.573445 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:08.573406    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf"]
Oct 30 20:36:08.577570 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:08.577544    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-48059e-6hgvf"]
Oct 30 20:36:09.529575 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:09.529543    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f" path="/var/lib/kubelet/pods/d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f/volumes"
Oct 30 20:36:09.658426 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:36:09.658393    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:36:15.840356 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:15.840325    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"]
Oct 30 20:36:15.840984 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:15.840602    5408 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt" podUID="9219bcf1-acbc-4725-95bd-791768bcbd0b" containerName="istio-proxy" containerID="cri-o://be132da32f12936eeabcb7426e6c90d6b3f216daca368b97cf98251a368b3cee" gracePeriod=30
Oct 30 20:36:21.045464 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.045426    5408 generic.go:356] "Generic (PLEG): container finished" podID="9219bcf1-acbc-4725-95bd-791768bcbd0b" containerID="be132da32f12936eeabcb7426e6c90d6b3f216daca368b97cf98251a368b3cee" exitCode=0
Oct 30 20:36:21.045824 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.045491    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt" event={"ID":"9219bcf1-acbc-4725-95bd-791768bcbd0b","Type":"ContainerDied","Data":"be132da32f12936eeabcb7426e6c90d6b3f216daca368b97cf98251a368b3cee"}
Oct 30 20:36:21.045824 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.045556    5408 scope.go:122] "RemoveContainer" containerID="c58908195b631eb03e47376acc87347b9a8b4410b061b753ed32599cd897b7fe"
Oct 30 20:36:21.078563 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.078526    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:36:21.179176 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.179110    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9219bcf1-acbc-4725-95bd-791768bcbd0b-kube-api-access-kk62q\" (UniqueName: \"kubernetes.io/projected/9219bcf1-acbc-4725-95bd-791768bcbd0b-kube-api-access-kk62q\") pod \"9219bcf1-acbc-4725-95bd-791768bcbd0b\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") "
Oct 30 20:36:21.179327 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.179195    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-data\") pod \"9219bcf1-acbc-4725-95bd-791768bcbd0b\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") "
Oct 30 20:36:21.179327 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.179271    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-workload-certs\") pod \"9219bcf1-acbc-4725-95bd-791768bcbd0b\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") "
Oct 30 20:36:21.179327 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.179305    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9219bcf1-acbc-4725-95bd-791768bcbd0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9219bcf1-acbc-4725-95bd-791768bcbd0b-istiod-ca-cert\") pod \"9219bcf1-acbc-4725-95bd-791768bcbd0b\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") "
Oct 30 20:36:21.179504 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.179342    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-envoy\") pod \"9219bcf1-acbc-4725-95bd-791768bcbd0b\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") "
Oct 30 20:36:21.179504 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.179376    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-podinfo\") pod \"9219bcf1-acbc-4725-95bd-791768bcbd0b\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") "
Oct 30 20:36:21.179504 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.179407    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-credential-socket\") pod \"9219bcf1-acbc-4725-95bd-791768bcbd0b\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") "
Oct 30 20:36:21.179504 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.179439    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-workload-socket\") pod \"9219bcf1-acbc-4725-95bd-791768bcbd0b\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") "
Oct 30 20:36:21.179504 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.179469    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-token\" (UniqueName: \"kubernetes.io/projected/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-token\") pod \"9219bcf1-acbc-4725-95bd-791768bcbd0b\" (UID: \"9219bcf1-acbc-4725-95bd-791768bcbd0b\") "
Oct 30 20:36:21.179763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.179511    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-data" pod "9219bcf1-acbc-4725-95bd-791768bcbd0b" (UID: "9219bcf1-acbc-4725-95bd-791768bcbd0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:36:21.179763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.179701    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9219bcf1-acbc-4725-95bd-791768bcbd0b-istiod-ca-cert" pod "9219bcf1-acbc-4725-95bd-791768bcbd0b" (UID: "9219bcf1-acbc-4725-95bd-791768bcbd0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:21.179763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.179700    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-credential-socket" pod "9219bcf1-acbc-4725-95bd-791768bcbd0b" (UID: "9219bcf1-acbc-4725-95bd-791768bcbd0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:36:21.179763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.179733    5408 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-data\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:21.179945 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.179811    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-workload-socket" pod "9219bcf1-acbc-4725-95bd-791768bcbd0b" (UID: "9219bcf1-acbc-4725-95bd-791768bcbd0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:36:21.180469 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.180097    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-workload-certs" pod "9219bcf1-acbc-4725-95bd-791768bcbd0b" (UID: "9219bcf1-acbc-4725-95bd-791768bcbd0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:36:21.181478 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.181451    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9219bcf1-acbc-4725-95bd-791768bcbd0b-kube-api-access-kk62q" pod "9219bcf1-acbc-4725-95bd-791768bcbd0b" (UID: "9219bcf1-acbc-4725-95bd-791768bcbd0b"). InnerVolumeSpecName "kube-api-access-kk62q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:21.181734 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.181713    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-token" pod "9219bcf1-acbc-4725-95bd-791768bcbd0b" (UID: "9219bcf1-acbc-4725-95bd-791768bcbd0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:21.181844 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.181822    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-envoy" pod "9219bcf1-acbc-4725-95bd-791768bcbd0b" (UID: "9219bcf1-acbc-4725-95bd-791768bcbd0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:36:21.181963 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.181939    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-podinfo" pod "9219bcf1-acbc-4725-95bd-791768bcbd0b" (UID: "9219bcf1-acbc-4725-95bd-791768bcbd0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:36:21.280619 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.280585    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk62q\" (UniqueName: \"kubernetes.io/projected/9219bcf1-acbc-4725-95bd-791768bcbd0b-kube-api-access-kk62q\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:21.280619 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.280619    5408 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-workload-certs\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:21.280781 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.280634    5408 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9219bcf1-acbc-4725-95bd-791768bcbd0b-istiod-ca-cert\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:21.280781 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.280647    5408 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-envoy\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:21.280781 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.280659    5408 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-podinfo\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:21.280781 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.280671    5408 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-credential-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:21.280781 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.280683    5408 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9219bcf1-acbc-4725-95bd-791768bcbd0b-workload-socket\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:21.280781 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:21.280695    5408 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9219bcf1-acbc-4725-95bd-791768bcbd0b-istio-token\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:22.051701 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:22.051667    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt" event={"ID":"9219bcf1-acbc-4725-95bd-791768bcbd0b","Type":"ContainerDied","Data":"bb0d56742f3c826e339a13861bd32b04d94380127bbe0ee67e063e866b9deae9"}
Oct 30 20:36:22.051701 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:22.051684    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"
Oct 30 20:36:22.052235 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:22.051723    5408 scope.go:122] "RemoveContainer" containerID="be132da32f12936eeabcb7426e6c90d6b3f216daca368b97cf98251a368b3cee"
Oct 30 20:36:22.072755 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:22.072734    5408 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"]
Oct 30 20:36:22.078972 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:22.078952    5408 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-48059e-openshift-default-779fc7bd9-mqmxt"]
Oct 30 20:36:23.531832 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:23.531791    5408 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9219bcf1-acbc-4725-95bd-791768bcbd0b" path="/var/lib/kubelet/pods/9219bcf1-acbc-4725-95bd-791768bcbd0b/volumes"
Oct 30 20:36:27.577931 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.577892    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5"]
Oct 30 20:36:27.578357 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.578270    5408 cpu_manager.go:414] "containerMap: removing container" podUID="9219bcf1-acbc-4725-95bd-791768bcbd0b" containerName="istio-proxy"
Oct 30 20:36:27.578357 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.578282    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9219bcf1-acbc-4725-95bd-791768bcbd0b" containerName="istio-proxy"
Oct 30 20:36:27.578357 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.578299    5408 cpu_manager.go:414] "containerMap: removing container" podUID="9219bcf1-acbc-4725-95bd-791768bcbd0b" containerName="istio-proxy"
Oct 30 20:36:27.578357 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.578304    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9219bcf1-acbc-4725-95bd-791768bcbd0b" containerName="istio-proxy"
Oct 30 20:36:27.578357 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.578329    5408 cpu_manager.go:414] "containerMap: removing container" podUID="d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f" containerName="revoke-keys"
Oct 30 20:36:27.578357 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.578335    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f" containerName="revoke-keys"
Oct 30 20:36:27.578572 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.578409    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a5bc2d-74b6-4fb4-a0fa-3787dee0450f" containerName="revoke-keys"
Oct 30 20:36:27.578572 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.578420    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="9219bcf1-acbc-4725-95bd-791768bcbd0b" containerName="istio-proxy"
Oct 30 20:36:27.578572 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.578429    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="9219bcf1-acbc-4725-95bd-791768bcbd0b" containerName="istio-proxy"
Oct 30 20:36:27.581646 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.581627    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5"
Oct 30 20:36:27.586114 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.586086    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:36:27.587541 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.587522    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-khrtr\""
Oct 30 20:36:27.590665 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.590631    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5"]
Oct 30 20:36:27.687789 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.687753    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8747203-33ab-4e4c-95bb-09f41d8ca921-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-t97p5\" (UID: \"a8747203-33ab-4e4c-95bb-09f41d8ca921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5"
Oct 30 20:36:27.687789 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.687790    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8747203-33ab-4e4c-95bb-09f41d8ca921-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-t97p5\" (UID: \"a8747203-33ab-4e4c-95bb-09f41d8ca921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5"
Oct 30 20:36:27.687997 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.687969    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b726q\" (UniqueName: \"kubernetes.io/projected/a8747203-33ab-4e4c-95bb-09f41d8ca921-kube-api-access-b726q\") pod \"maas-api-revoke-sub-f03629db-t97p5\" (UID: \"a8747203-33ab-4e4c-95bb-09f41d8ca921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5"
Oct 30 20:36:27.788451 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.788412    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b726q\" (UniqueName: \"kubernetes.io/projected/a8747203-33ab-4e4c-95bb-09f41d8ca921-kube-api-access-b726q\") pod \"maas-api-revoke-sub-f03629db-t97p5\" (UID: \"a8747203-33ab-4e4c-95bb-09f41d8ca921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5"
Oct 30 20:36:27.788638 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.788463    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8747203-33ab-4e4c-95bb-09f41d8ca921-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-t97p5\" (UID: \"a8747203-33ab-4e4c-95bb-09f41d8ca921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5"
Oct 30 20:36:27.788638 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.788488    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8747203-33ab-4e4c-95bb-09f41d8ca921-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-t97p5\" (UID: \"a8747203-33ab-4e4c-95bb-09f41d8ca921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5"
Oct 30 20:36:27.789217 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.789189    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8747203-33ab-4e4c-95bb-09f41d8ca921-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-t97p5\" (UID: \"a8747203-33ab-4e4c-95bb-09f41d8ca921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5"
Oct 30 20:36:27.790971 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.790941    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8747203-33ab-4e4c-95bb-09f41d8ca921-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-t97p5\" (UID: \"a8747203-33ab-4e4c-95bb-09f41d8ca921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5"
Oct 30 20:36:27.804508 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.804488    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b726q\" (UniqueName: \"kubernetes.io/projected/a8747203-33ab-4e4c-95bb-09f41d8ca921-kube-api-access-b726q\") pod \"maas-api-revoke-sub-f03629db-t97p5\" (UID: \"a8747203-33ab-4e4c-95bb-09f41d8ca921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5"
Oct 30 20:36:27.895811 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:27.895734    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5"
Oct 30 20:36:28.174676 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:28.174596    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5"]
Oct 30 20:36:28.177684 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:36:28.177658    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8747203_33ab_4e4c_95bb_09f41d8ca921.slice/crio-91cba9186202918a7aa8fc97b40e3257bc61a7043f533f5fefdea24c240c96f3 WatchSource:0}: Error finding container 91cba9186202918a7aa8fc97b40e3257bc61a7043f533f5fefdea24c240c96f3: Status 404 returned error can't find the container with id 91cba9186202918a7aa8fc97b40e3257bc61a7043f533f5fefdea24c240c96f3
Oct 30 20:36:29.094201 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:29.094165    5408 generic.go:356] "Generic (PLEG): container finished" podID="a8747203-33ab-4e4c-95bb-09f41d8ca921" containerID="4eb2cff1d6f17b6b354618402290b434f2082d81e5bd918f392d755096c0b6c5" exitCode=0
Oct 30 20:36:29.094577 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:29.094220    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5" event={"ID":"a8747203-33ab-4e4c-95bb-09f41d8ca921","Type":"ContainerDied","Data":"4eb2cff1d6f17b6b354618402290b434f2082d81e5bd918f392d755096c0b6c5"}
Oct 30 20:36:29.094577 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:29.094275    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5" event={"ID":"a8747203-33ab-4e4c-95bb-09f41d8ca921","Type":"ContainerStarted","Data":"91cba9186202918a7aa8fc97b40e3257bc61a7043f533f5fefdea24c240c96f3"}
Oct 30 20:36:30.228362 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:30.228336    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5"
Oct 30 20:36:30.313573 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:30.313531    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8747203-33ab-4e4c-95bb-09f41d8ca921-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8747203-33ab-4e4c-95bb-09f41d8ca921-maas-api-xks-ca\") pod \"a8747203-33ab-4e4c-95bb-09f41d8ca921\" (UID: \"a8747203-33ab-4e4c-95bb-09f41d8ca921\") "
Oct 30 20:36:30.313733 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:30.313620    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8747203-33ab-4e4c-95bb-09f41d8ca921-kube-api-access-b726q\" (UniqueName: \"kubernetes.io/projected/a8747203-33ab-4e4c-95bb-09f41d8ca921-kube-api-access-b726q\") pod \"a8747203-33ab-4e4c-95bb-09f41d8ca921\" (UID: \"a8747203-33ab-4e4c-95bb-09f41d8ca921\") "
Oct 30 20:36:30.313733 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:30.313686    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8747203-33ab-4e4c-95bb-09f41d8ca921-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8747203-33ab-4e4c-95bb-09f41d8ca921-maas-api-openshift-ca\") pod \"a8747203-33ab-4e4c-95bb-09f41d8ca921\" (UID: \"a8747203-33ab-4e4c-95bb-09f41d8ca921\") "
Oct 30 20:36:30.314067 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:30.314015    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8747203-33ab-4e4c-95bb-09f41d8ca921-maas-api-openshift-ca" pod "a8747203-33ab-4e4c-95bb-09f41d8ca921" (UID: "a8747203-33ab-4e4c-95bb-09f41d8ca921"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:30.315737 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:30.315712    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8747203-33ab-4e4c-95bb-09f41d8ca921-kube-api-access-b726q" pod "a8747203-33ab-4e4c-95bb-09f41d8ca921" (UID: "a8747203-33ab-4e4c-95bb-09f41d8ca921"). InnerVolumeSpecName "kube-api-access-b726q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:30.315823 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:30.315711    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8747203-33ab-4e4c-95bb-09f41d8ca921-maas-api-xks-ca" pod "a8747203-33ab-4e4c-95bb-09f41d8ca921" (UID: "a8747203-33ab-4e4c-95bb-09f41d8ca921"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:30.414891 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:30.414811    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8747203-33ab-4e4c-95bb-09f41d8ca921-maas-api-xks-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:30.414891 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:30.414839    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b726q\" (UniqueName: \"kubernetes.io/projected/a8747203-33ab-4e4c-95bb-09f41d8ca921-kube-api-access-b726q\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:30.414891 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:30.414849    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8747203-33ab-4e4c-95bb-09f41d8ca921-maas-api-openshift-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:31.104168 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:31.104132    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5"
Oct 30 20:36:31.104168 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:31.104152    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t97p5" event={"ID":"a8747203-33ab-4e4c-95bb-09f41d8ca921","Type":"ContainerDied","Data":"91cba9186202918a7aa8fc97b40e3257bc61a7043f533f5fefdea24c240c96f3"}
Oct 30 20:36:31.104393 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:31.104196    5408 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91cba9186202918a7aa8fc97b40e3257bc61a7043f533f5fefdea24c240c96f3"
Oct 30 20:36:39.659819 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:36:39.659777    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:36:42.619054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:42.616134    5408 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-j88mg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:36:42.619054 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:42.616221    5408 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-j88mg" podUID="e1be84f4-e5e8-4670-b67e-e3064f82f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:36:43.587119 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:43.587082    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9"]
Oct 30 20:36:43.587443 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:43.587429    5408 cpu_manager.go:414] "containerMap: removing container" podUID="a8747203-33ab-4e4c-95bb-09f41d8ca921" containerName="revoke-keys"
Oct 30 20:36:43.587497 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:43.587444    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8747203-33ab-4e4c-95bb-09f41d8ca921" containerName="revoke-keys"
Oct 30 20:36:43.587555 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:43.587523    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8747203-33ab-4e4c-95bb-09f41d8ca921" containerName="revoke-keys"
Oct 30 20:36:43.590671 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:43.590652    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9"
Oct 30 20:36:43.595762 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:43.595742    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:36:43.595871 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:43.595822    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-khrtr\""
Oct 30 20:36:43.600424 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:43.600401    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9"]
Oct 30 20:36:43.669374 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:43.669351    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnj76\" (UniqueName: \"kubernetes.io/projected/538915a1-15a3-4078-bf8f-0bec3e732d82-kube-api-access-tnj76\") pod \"maas-api-revoke-sub-0165d7d2-2wpm9\" (UID: \"538915a1-15a3-4078-bf8f-0bec3e732d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9"
Oct 30 20:36:43.669737 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:43.669402    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/538915a1-15a3-4078-bf8f-0bec3e732d82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-2wpm9\" (UID: \"538915a1-15a3-4078-bf8f-0bec3e732d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9"
Oct 30 20:36:43.669737 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:43.669425    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/538915a1-15a3-4078-bf8f-0bec3e732d82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-2wpm9\" (UID: \"538915a1-15a3-4078-bf8f-0bec3e732d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9"
Oct 30 20:36:43.770047 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:43.770004    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/538915a1-15a3-4078-bf8f-0bec3e732d82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-2wpm9\" (UID: \"538915a1-15a3-4078-bf8f-0bec3e732d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9"
Oct 30 20:36:43.770188 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:43.770064    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/538915a1-15a3-4078-bf8f-0bec3e732d82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-2wpm9\" (UID: \"538915a1-15a3-4078-bf8f-0bec3e732d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9"
Oct 30 20:36:43.770188 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:43.770118    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnj76\" (UniqueName: \"kubernetes.io/projected/538915a1-15a3-4078-bf8f-0bec3e732d82-kube-api-access-tnj76\") pod \"maas-api-revoke-sub-0165d7d2-2wpm9\" (UID: \"538915a1-15a3-4078-bf8f-0bec3e732d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9"
Oct 30 20:36:43.770732 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:43.770709    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/538915a1-15a3-4078-bf8f-0bec3e732d82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-2wpm9\" (UID: \"538915a1-15a3-4078-bf8f-0bec3e732d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9"
Oct 30 20:36:43.772401 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:43.772383    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/538915a1-15a3-4078-bf8f-0bec3e732d82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-2wpm9\" (UID: \"538915a1-15a3-4078-bf8f-0bec3e732d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9"
Oct 30 20:36:43.782871 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:43.782851    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnj76\" (UniqueName: \"kubernetes.io/projected/538915a1-15a3-4078-bf8f-0bec3e732d82-kube-api-access-tnj76\") pod \"maas-api-revoke-sub-0165d7d2-2wpm9\" (UID: \"538915a1-15a3-4078-bf8f-0bec3e732d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9"
Oct 30 20:36:43.904108 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:43.904017    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9"
Oct 30 20:36:44.182360 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:44.182274    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9"]
Oct 30 20:36:44.185403 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:36:44.185376    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538915a1_15a3_4078_bf8f_0bec3e732d82.slice/crio-a18bb69219c00565d64b2524c6dd1dd560d38d2998bf59083944ec6e50f9d083 WatchSource:0}: Error finding container a18bb69219c00565d64b2524c6dd1dd560d38d2998bf59083944ec6e50f9d083: Status 404 returned error can't find the container with id a18bb69219c00565d64b2524c6dd1dd560d38d2998bf59083944ec6e50f9d083
Oct 30 20:36:45.170763 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:45.170736    5408 generic.go:356] "Generic (PLEG): container finished" podID="538915a1-15a3-4078-bf8f-0bec3e732d82" containerID="701461b6a429e32bd3658211fdc056da431b77ba033ca63b2919a0712feb819b" exitCode=0
Oct 30 20:36:45.171129 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:45.170814    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9" event={"ID":"538915a1-15a3-4078-bf8f-0bec3e732d82","Type":"ContainerDied","Data":"701461b6a429e32bd3658211fdc056da431b77ba033ca63b2919a0712feb819b"}
Oct 30 20:36:45.171129 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:45.170850    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9" event={"ID":"538915a1-15a3-4078-bf8f-0bec3e732d82","Type":"ContainerStarted","Data":"a18bb69219c00565d64b2524c6dd1dd560d38d2998bf59083944ec6e50f9d083"}
Oct 30 20:36:46.297177 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:46.297151    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9"
Oct 30 20:36:46.392830 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:46.392801    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/538915a1-15a3-4078-bf8f-0bec3e732d82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/538915a1-15a3-4078-bf8f-0bec3e732d82-maas-api-xks-ca\") pod \"538915a1-15a3-4078-bf8f-0bec3e732d82\" (UID: \"538915a1-15a3-4078-bf8f-0bec3e732d82\") "
Oct 30 20:36:46.392999 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:46.392848    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/538915a1-15a3-4078-bf8f-0bec3e732d82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/538915a1-15a3-4078-bf8f-0bec3e732d82-maas-api-openshift-ca\") pod \"538915a1-15a3-4078-bf8f-0bec3e732d82\" (UID: \"538915a1-15a3-4078-bf8f-0bec3e732d82\") "
Oct 30 20:36:46.393087 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:46.393005    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/538915a1-15a3-4078-bf8f-0bec3e732d82-kube-api-access-tnj76\" (UniqueName: \"kubernetes.io/projected/538915a1-15a3-4078-bf8f-0bec3e732d82-kube-api-access-tnj76\") pod \"538915a1-15a3-4078-bf8f-0bec3e732d82\" (UID: \"538915a1-15a3-4078-bf8f-0bec3e732d82\") "
Oct 30 20:36:46.393209 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:46.393190    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/538915a1-15a3-4078-bf8f-0bec3e732d82-maas-api-openshift-ca" pod "538915a1-15a3-4078-bf8f-0bec3e732d82" (UID: "538915a1-15a3-4078-bf8f-0bec3e732d82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:46.393349 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:46.393336    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/538915a1-15a3-4078-bf8f-0bec3e732d82-maas-api-openshift-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:46.394924 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:46.394899    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/538915a1-15a3-4078-bf8f-0bec3e732d82-maas-api-xks-ca" pod "538915a1-15a3-4078-bf8f-0bec3e732d82" (UID: "538915a1-15a3-4078-bf8f-0bec3e732d82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:46.395083 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:46.394990    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/538915a1-15a3-4078-bf8f-0bec3e732d82-kube-api-access-tnj76" pod "538915a1-15a3-4078-bf8f-0bec3e732d82" (UID: "538915a1-15a3-4078-bf8f-0bec3e732d82"). InnerVolumeSpecName "kube-api-access-tnj76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:46.493844 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:46.493812    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/538915a1-15a3-4078-bf8f-0bec3e732d82-maas-api-xks-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:46.493844 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:46.493845    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnj76\" (UniqueName: \"kubernetes.io/projected/538915a1-15a3-4078-bf8f-0bec3e732d82-kube-api-access-tnj76\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:47.179993 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:47.179955    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9"
Oct 30 20:36:47.180264 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:47.179955    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2wpm9" event={"ID":"538915a1-15a3-4078-bf8f-0bec3e732d82","Type":"ContainerDied","Data":"a18bb69219c00565d64b2524c6dd1dd560d38d2998bf59083944ec6e50f9d083"}
Oct 30 20:36:47.180264 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:47.180126    5408 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a18bb69219c00565d64b2524c6dd1dd560d38d2998bf59083944ec6e50f9d083"
Oct 30 20:36:54.452427 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:54.452387    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk"]
Oct 30 20:36:54.452833 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:54.452807    5408 cpu_manager.go:414] "containerMap: removing container" podUID="538915a1-15a3-4078-bf8f-0bec3e732d82" containerName="revoke-keys"
Oct 30 20:36:54.452833 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:54.452821    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538915a1-15a3-4078-bf8f-0bec3e732d82" containerName="revoke-keys"
Oct 30 20:36:54.452914 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:54.452880    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="538915a1-15a3-4078-bf8f-0bec3e732d82" containerName="revoke-keys"
Oct 30 20:36:54.455965 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:54.455942    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk"
Oct 30 20:36:54.466620 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:54.466584    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk"]
Oct 30 20:36:54.487021 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:54.486990    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:36:54.487485 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:54.487129    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-khrtr\""
Oct 30 20:36:54.568000 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:54.567961    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12064363-48e5-4beb-9766-fb0960e0c7d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-mznpk\" (UID: \"12064363-48e5-4beb-9766-fb0960e0c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk"
Oct 30 20:36:54.568222 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:54.568021    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lck7p\" (UniqueName: \"kubernetes.io/projected/12064363-48e5-4beb-9766-fb0960e0c7d5-kube-api-access-lck7p\") pod \"maas-api-revoke-sub-6b19338f-mznpk\" (UID: \"12064363-48e5-4beb-9766-fb0960e0c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk"
Oct 30 20:36:54.568222 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:54.568149    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12064363-48e5-4beb-9766-fb0960e0c7d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-mznpk\" (UID: \"12064363-48e5-4beb-9766-fb0960e0c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk"
Oct 30 20:36:54.669067 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:54.669007    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lck7p\" (UniqueName: \"kubernetes.io/projected/12064363-48e5-4beb-9766-fb0960e0c7d5-kube-api-access-lck7p\") pod \"maas-api-revoke-sub-6b19338f-mznpk\" (UID: \"12064363-48e5-4beb-9766-fb0960e0c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk"
Oct 30 20:36:54.669067 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:54.669077    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12064363-48e5-4beb-9766-fb0960e0c7d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-mznpk\" (UID: \"12064363-48e5-4beb-9766-fb0960e0c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk"
Oct 30 20:36:54.669366 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:54.669128    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12064363-48e5-4beb-9766-fb0960e0c7d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-mznpk\" (UID: \"12064363-48e5-4beb-9766-fb0960e0c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk"
Oct 30 20:36:54.669762 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:54.669739    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12064363-48e5-4beb-9766-fb0960e0c7d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-mznpk\" (UID: \"12064363-48e5-4beb-9766-fb0960e0c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk"
Oct 30 20:36:54.671595 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:54.671572    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12064363-48e5-4beb-9766-fb0960e0c7d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-mznpk\" (UID: \"12064363-48e5-4beb-9766-fb0960e0c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk"
Oct 30 20:36:54.684619 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:54.684593    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lck7p\" (UniqueName: \"kubernetes.io/projected/12064363-48e5-4beb-9766-fb0960e0c7d5-kube-api-access-lck7p\") pod \"maas-api-revoke-sub-6b19338f-mznpk\" (UID: \"12064363-48e5-4beb-9766-fb0960e0c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk"
Oct 30 20:36:54.770370 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:54.770330    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk"
Oct 30 20:36:55.044878 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:55.044798    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk"]
Oct 30 20:36:55.048119 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:36:55.048087    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12064363_48e5_4beb_9766_fb0960e0c7d5.slice/crio-d60b5f8e2008b7252d2bb7093a3b4087276f391fef9de520bccd87ae33c11f4a WatchSource:0}: Error finding container d60b5f8e2008b7252d2bb7093a3b4087276f391fef9de520bccd87ae33c11f4a: Status 404 returned error can't find the container with id d60b5f8e2008b7252d2bb7093a3b4087276f391fef9de520bccd87ae33c11f4a
Oct 30 20:36:55.215804 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:55.215773    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk" event={"ID":"12064363-48e5-4beb-9766-fb0960e0c7d5","Type":"ContainerStarted","Data":"d60b5f8e2008b7252d2bb7093a3b4087276f391fef9de520bccd87ae33c11f4a"}
Oct 30 20:36:56.221405 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:56.221312    5408 generic.go:356] "Generic (PLEG): container finished" podID="12064363-48e5-4beb-9766-fb0960e0c7d5" containerID="2e3157fb041e370fbeec8d70e8e7ac77dcd525f77a045aeb67568a057bd20244" exitCode=0
Oct 30 20:36:56.221405 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:56.221385    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk" event={"ID":"12064363-48e5-4beb-9766-fb0960e0c7d5","Type":"ContainerDied","Data":"2e3157fb041e370fbeec8d70e8e7ac77dcd525f77a045aeb67568a057bd20244"}
Oct 30 20:36:57.353538 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:57.353513    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk"
Oct 30 20:36:57.397542 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:57.397508    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12064363-48e5-4beb-9766-fb0960e0c7d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12064363-48e5-4beb-9766-fb0960e0c7d5-maas-api-xks-ca\") pod \"12064363-48e5-4beb-9766-fb0960e0c7d5\" (UID: \"12064363-48e5-4beb-9766-fb0960e0c7d5\") "
Oct 30 20:36:57.397737 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:57.397579    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12064363-48e5-4beb-9766-fb0960e0c7d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12064363-48e5-4beb-9766-fb0960e0c7d5-maas-api-openshift-ca\") pod \"12064363-48e5-4beb-9766-fb0960e0c7d5\" (UID: \"12064363-48e5-4beb-9766-fb0960e0c7d5\") "
Oct 30 20:36:57.397737 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:57.397639    5408 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12064363-48e5-4beb-9766-fb0960e0c7d5-kube-api-access-lck7p\" (UniqueName: \"kubernetes.io/projected/12064363-48e5-4beb-9766-fb0960e0c7d5-kube-api-access-lck7p\") pod \"12064363-48e5-4beb-9766-fb0960e0c7d5\" (UID: \"12064363-48e5-4beb-9766-fb0960e0c7d5\") "
Oct 30 20:36:57.398022 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:57.397995    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12064363-48e5-4beb-9766-fb0960e0c7d5-maas-api-openshift-ca" pod "12064363-48e5-4beb-9766-fb0960e0c7d5" (UID: "12064363-48e5-4beb-9766-fb0960e0c7d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:57.399749 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:57.399712    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12064363-48e5-4beb-9766-fb0960e0c7d5-maas-api-xks-ca" pod "12064363-48e5-4beb-9766-fb0960e0c7d5" (UID: "12064363-48e5-4beb-9766-fb0960e0c7d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:57.399845 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:57.399769    5408 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12064363-48e5-4beb-9766-fb0960e0c7d5-kube-api-access-lck7p" pod "12064363-48e5-4beb-9766-fb0960e0c7d5" (UID: "12064363-48e5-4beb-9766-fb0960e0c7d5"). InnerVolumeSpecName "kube-api-access-lck7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:57.498331 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:57.498300    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12064363-48e5-4beb-9766-fb0960e0c7d5-maas-api-xks-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:57.498331 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:57.498333    5408 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12064363-48e5-4beb-9766-fb0960e0c7d5-maas-api-openshift-ca\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:57.498556 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:57.498343    5408 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lck7p\" (UniqueName: \"kubernetes.io/projected/12064363-48e5-4beb-9766-fb0960e0c7d5-kube-api-access-lck7p\") on node \"ip-10-0-141-19.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:58.230420 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:58.230387    5408 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk"
Oct 30 20:36:58.230602 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:58.230393    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mznpk" event={"ID":"12064363-48e5-4beb-9766-fb0960e0c7d5","Type":"ContainerDied","Data":"d60b5f8e2008b7252d2bb7093a3b4087276f391fef9de520bccd87ae33c11f4a"}
Oct 30 20:36:58.230602 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:36:58.230512    5408 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d60b5f8e2008b7252d2bb7093a3b4087276f391fef9de520bccd87ae33c11f4a"
Oct 30 20:37:09.659165 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:37:09.659129    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:37:17.331733 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:17.331703    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-28gd4_57afffa1-1703-46a7-829e-241810a74b2a/manager/0.log"
Oct 30 20:37:17.588758 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:17.588683    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df954cb67-w2ql2_67ef2465-c1a0-4950-a13e-80170dcf1928/manager/0.log"
Oct 30 20:37:18.218956 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:18.218926    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-px62s_d6bc6fc8-3673-4fcd-8125-fa63a3ce034a/proxy/0.log"
Oct 30 20:37:19.378323 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:19.378292    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-2wpm9_538915a1-15a3-4078-bf8f-0bec3e732d82/revoke-keys/0.log"
Oct 30 20:37:20.112660 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:20.112627    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-mznpk_12064363-48e5-4beb-9766-fb0960e0c7d5/revoke-keys/0.log"
Oct 30 20:37:20.289342 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:20.289313    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-t97p5_a8747203-33ab-4e4c-95bb-09f41d8ca921/revoke-keys/0.log"
Oct 30 20:37:20.461310 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:20.461230    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-z5bc2_50dfdefa-ac3c-4f7e-a440-d9669a993fb5/postgres/0.log"
Oct 30 20:37:24.493445 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:24.493412    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8w9xg_61adfc21-f356-4abd-bd63-5f690ec2e708/discovery/0.log"
Oct 30 20:37:24.880667 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:24.880580    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d87896589-x8jvc_59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb/kube-auth-proxy/0.log"
Oct 30 20:37:25.646292 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:25.646257    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-9nv8z_10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b/istio-proxy/0.log"
Oct 30 20:37:26.018525 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:26.018498    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-j88mg_e1be84f4-e5e8-4670-b67e-e3064f82f544/istio-proxy/1.log"
Oct 30 20:37:26.748439 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:26.748408    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-zgpln_cfc905b2-d5e2-4a89-9bbb-435393789d7f/istio-proxy/1.log"
Oct 30 20:37:28.499374 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:28.499340    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-j88mg_e1be84f4-e5e8-4670-b67e-e3064f82f544/istio-proxy/0.log"
Oct 30 20:37:28.880785 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:28.880705    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-zgpln_cfc905b2-d5e2-4a89-9bbb-435393789d7f/istio-proxy/0.log"
Oct 30 20:37:39.659487 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:37:39.659451    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:37:41.618699 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:41.618663    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-28gd4_57afffa1-1703-46a7-829e-241810a74b2a/manager/0.log"
Oct 30 20:37:41.839201 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:41.839168    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df954cb67-w2ql2_67ef2465-c1a0-4950-a13e-80170dcf1928/manager/0.log"
Oct 30 20:37:42.451512 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:42.451477    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-px62s_d6bc6fc8-3673-4fcd-8125-fa63a3ce034a/proxy/0.log"
Oct 30 20:37:43.385118 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:43.385092    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-2wpm9_538915a1-15a3-4078-bf8f-0bec3e732d82/revoke-keys/0.log"
Oct 30 20:37:43.930641 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:43.930610    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-mznpk_12064363-48e5-4beb-9766-fb0960e0c7d5/revoke-keys/0.log"
Oct 30 20:37:44.113481 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:44.113452    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-t97p5_a8747203-33ab-4e4c-95bb-09f41d8ca921/revoke-keys/0.log"
Oct 30 20:37:44.286383 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:44.286356    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-z5bc2_50dfdefa-ac3c-4f7e-a440-d9669a993fb5/postgres/0.log"
Oct 30 20:37:47.467111 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:47.467082    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8w9xg_61adfc21-f356-4abd-bd63-5f690ec2e708/discovery/0.log"
Oct 30 20:37:47.841999 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:47.841969    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d87896589-x8jvc_59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb/kube-auth-proxy/0.log"
Oct 30 20:37:48.601682 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:48.601647    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-9nv8z_10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b/istio-proxy/0.log"
Oct 30 20:37:48.984673 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:48.984596    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-j88mg_e1be84f4-e5e8-4670-b67e-e3064f82f544/istio-proxy/1.log"
Oct 30 20:37:49.731682 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:37:49.731635    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-zgpln_cfc905b2-d5e2-4a89-9bbb-435393789d7f/istio-proxy/1.log"
Oct 30 20:38:09.659632 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:38:09.659599    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:38:30.452396 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:38:30.452309    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df954cb67-w2ql2_67ef2465-c1a0-4950-a13e-80170dcf1928/manager/0.log"
Oct 30 20:38:30.722380 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:38:30.722292    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df954cb67-w2ql2_67ef2465-c1a0-4950-a13e-80170dcf1928/manager/0.log"
Oct 30 20:38:39.660069 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:38:39.660012    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:39:09.659262 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:39:09.659227    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:39:18.347551 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:18.347518    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df954cb67-w2ql2_67ef2465-c1a0-4950-a13e-80170dcf1928/manager/0.log"
Oct 30 20:39:18.602916 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:18.602846    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6df954cb67-w2ql2_67ef2465-c1a0-4950-a13e-80170dcf1928/manager/0.log"
Oct 30 20:39:19.875330 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:19.875297    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rzbpr/must-gather-jgjvm"]
Oct 30 20:39:19.875721 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:19.875627    5408 cpu_manager.go:414] "containerMap: removing container" podUID="12064363-48e5-4beb-9766-fb0960e0c7d5" containerName="revoke-keys"
Oct 30 20:39:19.875721 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:19.875638    5408 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12064363-48e5-4beb-9766-fb0960e0c7d5" containerName="revoke-keys"
Oct 30 20:39:19.875721 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:19.875706    5408 memory_manager.go:365] "RemoveStaleState removing state" podUID="12064363-48e5-4beb-9766-fb0960e0c7d5" containerName="revoke-keys"
Oct 30 20:39:19.878837 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:19.878819    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzbpr/must-gather-jgjvm"
Oct 30 20:39:19.886458 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:19.886436    5408 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rzbpr\"/\"default-dockercfg-czcz7\""
Oct 30 20:39:19.886583 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:19.886518    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rzbpr\"/\"kube-root-ca.crt\""
Oct 30 20:39:19.886583 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:19.886556    5408 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rzbpr\"/\"openshift-service-ca.crt\""
Oct 30 20:39:19.919303 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:19.916331    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rzbpr/must-gather-jgjvm"]
Oct 30 20:39:20.054863 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:20.054825    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpkfr\" (UniqueName: \"kubernetes.io/projected/44b70b5e-b518-4c90-96e7-be5a23e78fc0-kube-api-access-kpkfr\") pod \"must-gather-jgjvm\" (UID: \"44b70b5e-b518-4c90-96e7-be5a23e78fc0\") " pod="openshift-must-gather-rzbpr/must-gather-jgjvm"
Oct 30 20:39:20.055091 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:20.054961    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/44b70b5e-b518-4c90-96e7-be5a23e78fc0-must-gather-output\") pod \"must-gather-jgjvm\" (UID: \"44b70b5e-b518-4c90-96e7-be5a23e78fc0\") " pod="openshift-must-gather-rzbpr/must-gather-jgjvm"
Oct 30 20:39:20.156071 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:20.155961    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/44b70b5e-b518-4c90-96e7-be5a23e78fc0-must-gather-output\") pod \"must-gather-jgjvm\" (UID: \"44b70b5e-b518-4c90-96e7-be5a23e78fc0\") " pod="openshift-must-gather-rzbpr/must-gather-jgjvm"
Oct 30 20:39:20.156071 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:20.156023    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpkfr\" (UniqueName: \"kubernetes.io/projected/44b70b5e-b518-4c90-96e7-be5a23e78fc0-kube-api-access-kpkfr\") pod \"must-gather-jgjvm\" (UID: \"44b70b5e-b518-4c90-96e7-be5a23e78fc0\") " pod="openshift-must-gather-rzbpr/must-gather-jgjvm"
Oct 30 20:39:20.156334 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:20.156313    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/44b70b5e-b518-4c90-96e7-be5a23e78fc0-must-gather-output\") pod \"must-gather-jgjvm\" (UID: \"44b70b5e-b518-4c90-96e7-be5a23e78fc0\") " pod="openshift-must-gather-rzbpr/must-gather-jgjvm"
Oct 30 20:39:20.172728 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:20.172697    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpkfr\" (UniqueName: \"kubernetes.io/projected/44b70b5e-b518-4c90-96e7-be5a23e78fc0-kube-api-access-kpkfr\") pod \"must-gather-jgjvm\" (UID: \"44b70b5e-b518-4c90-96e7-be5a23e78fc0\") " pod="openshift-must-gather-rzbpr/must-gather-jgjvm"
Oct 30 20:39:20.192512 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:20.192490    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzbpr/must-gather-jgjvm"
Oct 30 20:39:20.468949 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:20.468922    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rzbpr/must-gather-jgjvm"]
Oct 30 20:39:20.471436 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:39:20.471408    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44b70b5e_b518_4c90_96e7_be5a23e78fc0.slice/crio-54ba603e3c75b2752fa21cdf0ffb2b93b58a2a818c69e0082a7e988d65f532ea WatchSource:0}: Error finding container 54ba603e3c75b2752fa21cdf0ffb2b93b58a2a818c69e0082a7e988d65f532ea: Status 404 returned error can't find the container with id 54ba603e3c75b2752fa21cdf0ffb2b93b58a2a818c69e0082a7e988d65f532ea
Oct 30 20:39:20.473222 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:20.473205    5408 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:39:20.846212 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:20.846180    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzbpr/must-gather-jgjvm" event={"ID":"44b70b5e-b518-4c90-96e7-be5a23e78fc0","Type":"ContainerStarted","Data":"54ba603e3c75b2752fa21cdf0ffb2b93b58a2a818c69e0082a7e988d65f532ea"}
Oct 30 20:39:21.853529 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:21.853493    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzbpr/must-gather-jgjvm" event={"ID":"44b70b5e-b518-4c90-96e7-be5a23e78fc0","Type":"ContainerStarted","Data":"70b878063d067ee954449ea963030297c34a05f80f8795cbbfba355ad839fac5"}
Oct 30 20:39:21.854025 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:21.853543    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzbpr/must-gather-jgjvm" event={"ID":"44b70b5e-b518-4c90-96e7-be5a23e78fc0","Type":"ContainerStarted","Data":"c3f599e4e8deca9490ab597f45db51421a566cb83c2b0d0b69fb75ab2dad2f5e"}
Oct 30 20:39:21.872300 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:21.872253    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rzbpr/must-gather-jgjvm" podStartSLOduration=2.107490773 podStartE2EDuration="2.872239643s" totalImagesPullingTime="764.74887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:21.871533836 +0000 UTC m=+4116.455803453" watchObservedRunningTime="2026-10-30 20:39:21.872239643 +0000 UTC m=+4116.456509259"
Oct 30 20:39:23.587912 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:23.587869    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d2hv2_8013e357-0fa2-4360-924b-18cbfe588b5c/global-pull-secret-syncer/0.log"
Oct 30 20:39:23.905933 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:23.905846    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kspfd_23de5b1c-4825-43ed-bff8-cdab8789989e/konnectivity-agent/0.log"
Oct 30 20:39:24.119777 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:24.119742    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-19.us-east-2.compute.internal_7dbf81fa0f84f4c1de7386fb2838c009/haproxy/0.log"
Oct 30 20:39:32.395294 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:32.395265    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-ldlt7_a341b7ef-207b-4443-b01e-9fcbc2ac0075/monitoring-plugin/0.log"
Oct 30 20:39:32.598817 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:32.598777    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rb876_0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf/node-exporter/0.log"
Oct 30 20:39:32.643149 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:32.643111    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rb876_0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf/kube-rbac-proxy/0.log"
Oct 30 20:39:32.684730 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:32.684658    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rb876_0fbc20d7-e495-4d90-8c4c-a2fcc76a0baf/init-textfile/0.log"
Oct 30 20:39:33.018176 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.018142    5408 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"]
Oct 30 20:39:33.026200 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.026173    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:33.035161 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.034991    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"]
Oct 30 20:39:33.190230 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.190195    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b880ad9-2477-4df1-8271-6aeabaa03d0a-sys\") pod \"perf-node-gather-daemonset-mb9rq\" (UID: \"4b880ad9-2477-4df1-8271-6aeabaa03d0a\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:33.190436 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.190238    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b880ad9-2477-4df1-8271-6aeabaa03d0a-proc\") pod \"perf-node-gather-daemonset-mb9rq\" (UID: \"4b880ad9-2477-4df1-8271-6aeabaa03d0a\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:33.190436 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.190263    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b880ad9-2477-4df1-8271-6aeabaa03d0a-podres\") pod \"perf-node-gather-daemonset-mb9rq\" (UID: \"4b880ad9-2477-4df1-8271-6aeabaa03d0a\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:33.190436 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.190426    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b880ad9-2477-4df1-8271-6aeabaa03d0a-lib-modules\") pod \"perf-node-gather-daemonset-mb9rq\" (UID: \"4b880ad9-2477-4df1-8271-6aeabaa03d0a\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:33.190580 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.190467    5408 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp5wv\" (UniqueName: \"kubernetes.io/projected/4b880ad9-2477-4df1-8271-6aeabaa03d0a-kube-api-access-gp5wv\") pod \"perf-node-gather-daemonset-mb9rq\" (UID: \"4b880ad9-2477-4df1-8271-6aeabaa03d0a\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:33.256209 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.256169    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-9s95l_008928f8-b2ce-4f12-9681-21136d802ff6/prometheus-operator/0.log"
Oct 30 20:39:33.291782 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.291686    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b880ad9-2477-4df1-8271-6aeabaa03d0a-lib-modules\") pod \"perf-node-gather-daemonset-mb9rq\" (UID: \"4b880ad9-2477-4df1-8271-6aeabaa03d0a\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:33.291782 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.291741    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5wv\" (UniqueName: \"kubernetes.io/projected/4b880ad9-2477-4df1-8271-6aeabaa03d0a-kube-api-access-gp5wv\") pod \"perf-node-gather-daemonset-mb9rq\" (UID: \"4b880ad9-2477-4df1-8271-6aeabaa03d0a\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:33.291782 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.291784    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b880ad9-2477-4df1-8271-6aeabaa03d0a-sys\") pod \"perf-node-gather-daemonset-mb9rq\" (UID: \"4b880ad9-2477-4df1-8271-6aeabaa03d0a\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:33.292115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.291812    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b880ad9-2477-4df1-8271-6aeabaa03d0a-proc\") pod \"perf-node-gather-daemonset-mb9rq\" (UID: \"4b880ad9-2477-4df1-8271-6aeabaa03d0a\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:33.292115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.291833    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b880ad9-2477-4df1-8271-6aeabaa03d0a-lib-modules\") pod \"perf-node-gather-daemonset-mb9rq\" (UID: \"4b880ad9-2477-4df1-8271-6aeabaa03d0a\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:33.292115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.291878    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b880ad9-2477-4df1-8271-6aeabaa03d0a-proc\") pod \"perf-node-gather-daemonset-mb9rq\" (UID: \"4b880ad9-2477-4df1-8271-6aeabaa03d0a\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:33.292115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.291886    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b880ad9-2477-4df1-8271-6aeabaa03d0a-sys\") pod \"perf-node-gather-daemonset-mb9rq\" (UID: \"4b880ad9-2477-4df1-8271-6aeabaa03d0a\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:33.292115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.291943    5408 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b880ad9-2477-4df1-8271-6aeabaa03d0a-podres\") pod \"perf-node-gather-daemonset-mb9rq\" (UID: \"4b880ad9-2477-4df1-8271-6aeabaa03d0a\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:33.292115 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.292047    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b880ad9-2477-4df1-8271-6aeabaa03d0a-podres\") pod \"perf-node-gather-daemonset-mb9rq\" (UID: \"4b880ad9-2477-4df1-8271-6aeabaa03d0a\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:33.301783 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.301754    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-9s95l_008928f8-b2ce-4f12-9681-21136d802ff6/kube-rbac-proxy/0.log"
Oct 30 20:39:33.306439 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.306408    5408 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp5wv\" (UniqueName: \"kubernetes.io/projected/4b880ad9-2477-4df1-8271-6aeabaa03d0a-kube-api-access-gp5wv\") pod \"perf-node-gather-daemonset-mb9rq\" (UID: \"4b880ad9-2477-4df1-8271-6aeabaa03d0a\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:33.343794 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.343762    5408 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:33.387051 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.386249    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f8b7447-wnrwb_8de260aa-48d7-4d43-9c07-458a5d3db82d/telemeter-client/0.log"
Oct 30 20:39:33.425855 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.425823    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f8b7447-wnrwb_8de260aa-48d7-4d43-9c07-458a5d3db82d/reload/0.log"
Oct 30 20:39:33.461947 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.461919    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f8b7447-wnrwb_8de260aa-48d7-4d43-9c07-458a5d3db82d/kube-rbac-proxy/0.log"
Oct 30 20:39:33.508018 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.507979    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67c75b6564-g8dq2_e0d19f3a-000a-4911-96dd-cbf24e784627/thanos-query/0.log"
Oct 30 20:39:33.540829 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.540797    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67c75b6564-g8dq2_e0d19f3a-000a-4911-96dd-cbf24e784627/kube-rbac-proxy-web/0.log"
Oct 30 20:39:33.575161 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.575084    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67c75b6564-g8dq2_e0d19f3a-000a-4911-96dd-cbf24e784627/kube-rbac-proxy/0.log"
Oct 30 20:39:33.611080 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.611048    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67c75b6564-g8dq2_e0d19f3a-000a-4911-96dd-cbf24e784627/prom-label-proxy/0.log"
Oct 30 20:39:33.641560 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.640339    5408 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"]
Oct 30 20:39:33.643629 ip-10-0-141-19 kubenswrapper[5408]: W1030 20:39:33.643600    5408 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4b880ad9_2477_4df1_8271_6aeabaa03d0a.slice/crio-3e7eaf26cf0e525cb7348bc03437e15ea2987381033bd3d812977b44a983631d WatchSource:0}: Error finding container 3e7eaf26cf0e525cb7348bc03437e15ea2987381033bd3d812977b44a983631d: Status 404 returned error can't find the container with id 3e7eaf26cf0e525cb7348bc03437e15ea2987381033bd3d812977b44a983631d
Oct 30 20:39:33.648463 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.648446    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67c75b6564-g8dq2_e0d19f3a-000a-4911-96dd-cbf24e784627/kube-rbac-proxy-rules/0.log"
Oct 30 20:39:33.692574 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.692417    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67c75b6564-g8dq2_e0d19f3a-000a-4911-96dd-cbf24e784627/kube-rbac-proxy-metrics/0.log"
Oct 30 20:39:33.950597 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.950506    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq" event={"ID":"4b880ad9-2477-4df1-8271-6aeabaa03d0a","Type":"ContainerStarted","Data":"1b33ca1220ad565ca9ad47ca92dc4a3f2e88f0eb34fbd487696069e5f3cdf16b"}
Oct 30 20:39:33.950760 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.950708    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:33.950760 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.950744    5408 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq" event={"ID":"4b880ad9-2477-4df1-8271-6aeabaa03d0a","Type":"ContainerStarted","Data":"3e7eaf26cf0e525cb7348bc03437e15ea2987381033bd3d812977b44a983631d"}
Oct 30 20:39:33.970544 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:33.970480    5408 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq" podStartSLOduration=1.970462469 podStartE2EDuration="1.970462469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:33.968936367 +0000 UTC m=+4128.553205977" watchObservedRunningTime="2026-10-30 20:39:33.970462469 +0000 UTC m=+4128.554732087"
Oct 30 20:39:36.126306 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:36.126254    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-857b84547b-9wx6x_9af4bc7e-b468-47a8-baee-d0c293f6f728/console/0.log"
Oct 30 20:39:38.089879 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:38.089848    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dkkwh_c4e933e1-4e6d-4a26-bb7d-68f366e02ac5/dns/0.log"
Oct 30 20:39:38.129278 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:38.129249    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dkkwh_c4e933e1-4e6d-4a26-bb7d-68f366e02ac5/kube-rbac-proxy/0.log"
Oct 30 20:39:38.268901 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:38.268865    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gmrbv_c2836676-0428-4f2f-a801-0a25df87aee2/dns-node-resolver/0.log"
Oct 30 20:39:39.139088 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:39.137998    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h5m52_53bd98ea-ea87-484a-b7b1-8c9070fc9b25/node-ca/0.log"
Oct 30 20:39:39.680529 ip-10-0-141-19 kubenswrapper[5408]: E1030 20:39:39.680493    5408 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rkc6x" containerName="extractor"
Oct 30 20:39:39.973102 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:39.972998    5408 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-mb9rq"
Oct 30 20:39:42.886105 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:42.886046    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8w9xg_61adfc21-f356-4abd-bd63-5f690ec2e708/discovery/0.log"
Oct 30 20:39:43.398935 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:43.398895    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d87896589-x8jvc_59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb/kube-auth-proxy/0.log"
Oct 30 20:39:43.588598 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:43.588568    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-9nv8z_10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b/istio-proxy/0.log"
Oct 30 20:39:43.666362 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:43.666289    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-j88mg_e1be84f4-e5e8-4670-b67e-e3064f82f544/istio-proxy/0.log"
Oct 30 20:39:43.666362 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:43.666327    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-j88mg_e1be84f4-e5e8-4670-b67e-e3064f82f544/istio-proxy/1.log"
Oct 30 20:39:43.829149 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:43.829123    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-zgpln_cfc905b2-d5e2-4a89-9bbb-435393789d7f/istio-proxy/1.log"
Oct 30 20:39:43.829400 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:43.829326    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-zgpln_cfc905b2-d5e2-4a89-9bbb-435393789d7f/istio-proxy/0.log"
Oct 30 20:39:46.544218 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:46.544187    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2c2l4_0de2915a-789c-4acc-84a6-c61987498b62/serve-healthcheck-canary/0.log"
Oct 30 20:39:46.753144 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:46.753115    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8w9xg_61adfc21-f356-4abd-bd63-5f690ec2e708/discovery/0.log"
Oct 30 20:39:46.784240 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:46.784207    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d87896589-x8jvc_59f8e2b6-46cc-487a-9c8b-20fb1c7f63eb/kube-auth-proxy/0.log"
Oct 30 20:39:46.867246 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:46.867168    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-9nv8z_10afc8d2-2e5e-4c2f-a3a3-2cb9d2bcf80b/istio-proxy/0.log"
Oct 30 20:39:46.908341 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:46.908315    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-j88mg_e1be84f4-e5e8-4670-b67e-e3064f82f544/istio-proxy/0.log"
Oct 30 20:39:46.908341 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:46.908347    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-j88mg_e1be84f4-e5e8-4670-b67e-e3064f82f544/istio-proxy/1.log"
Oct 30 20:39:46.988756 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:46.988728    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-zgpln_cfc905b2-d5e2-4a89-9bbb-435393789d7f/istio-proxy/1.log"
Oct 30 20:39:46.989046 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:46.989006    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-zgpln_cfc905b2-d5e2-4a89-9bbb-435393789d7f/istio-proxy/0.log"
Oct 30 20:39:47.358316 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:47.358265    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rkc6x_45ccae2d-1b91-4bbe-9598-94ad740116ec/kube-rbac-proxy/0.log"
Oct 30 20:39:47.396953 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:47.396901    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rkc6x_45ccae2d-1b91-4bbe-9598-94ad740116ec/exporter/0.log"
Oct 30 20:39:47.428294 ip-10-0-141-19 kubenswrapper[5408]: I1030 20:39:47.428265    5408 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rkc6x_45ccae2d-1b91-4bbe-9598-94ad740116ec/extractor/0.log"